When things started out with ChatGPT 3 years ago (It has only been three years ago?!), many people were pretending not to use it.

As things progressed, quite quickly, engineers would “leave it in the middle” whether they were using AI and how.

Now that we are seeing tooling for LLM-assisted code development gain popularity with copilot, cursor and windsurf, there’s no more dilly dallying around it:

Most all engineers are using AI in their workflows.

So, why do our code bases then not reflect this reality yet?

Now that we got that out of the way, let’s get into it.

A new era of software engineering: Prompt Engineering?

Two weeks ago, Andrej Karpathy had some fascinating insights in his recent Y-combinator talk on how “software is changing again”.

In one part he was describing how the code base for self-driving evolved in tesla into roughly three paradigms:

Software 1.0 — Classical software.

Software 2.0 — The training and weighing of neural nets.

Software 3.0 — Prompts to the trained neural nets.

It has really struck me that the archetype of the codebase as a whole was changing and I think we can draw parallels into code bases as a whole in the industry in the years to come.

That got me to think however: what IS the best way to deal with Software 3.0 or prompts in a code base?

Everyone is talking to LLMs in isolation

Let’s take a step back and zoom out a bit.

Let’s say we have a code base for our team and we have 10 engineers that are prompting the LLM on the same code base to develop features for it.

How much time do you spend on refining your prompts and explaining your code base to an LLM?

Anyone who has put serious time into these will say: “A lot”.

Obviously, this improves with:

• Every new iteration of AI models

• Larger context windows

• Smart tricks like code editors indexing your code base to understand it’s structure

— but that doesn’t mean that it will tackle the problem at hand in the most efficient way given current constraints.

In my own testing of this, I’ve seen this can go wrong in a couple of ways in practise. Some examples:

• There are multiple legacy patterns in the code base that need to be ignored for certain pieces of code, but not for others.

• The LLM (like a human) might try to take a shortcut and write a one-off solution that is not congruent with the pattern within the code base.

• Having more irrelevant context decreases the amount of relevant context available for my query

• Too much context can confuse models

The cost of this is quite significant.

If we know for a fact that engineers will be using LLMs for development
if we know that engineering hours are expensive
Then we need to acknowledge the fact that if we don’t optimize for their interactions with an LLM we are losing valuable resources in time and effort.

hours wasted prompting LLMs = Wasted Engineering hours = 💸💸💸

Do you really want all the engineers in a team reinventing the wheel every time when it comes to how they work with the LLM on more repetitive and predictable parts of a code base?

There has been some development in the field to fine-tune these sorts of issues.

Tools like cursor introduce the concept of rules which are reusable instructions that can be applied selectively based on user input, location of a file in a code base and more. Others like GitHub implemented organization wide knowledge bases that can be used by LLM coding agents in the IDE.

There is still work to be done on how knowledge and prompts will be organized in code bases — that is a given — but the concept of how to actually properly build and integrate this knowledge is a topic that will stay relevant no matter what technology will be deployed.

With that, there is a fundamental psychological change needed when it comes to developing and maintaining code as a team which I would like to coin as “knowledge driven development”.

Knowledge driven development is a development paradigm that prioritizes the structuring, curating and generation of context for AI and for humans in a code base.

For the purposes of this article, I will (mostly) not be going over knowledge basis but rather human crafted prompts and context.

Well then, let’s take a look at what that might look like in practise and the potential longer term ramifications of that.

Documentation re-envisioned: Prompt context as documentation

Once we reach a certain complexity in a code base, we all know that “just reading the code” to understand how everything ties together becomes increasingly complicated.

There is also the fact that as a code base grows in importance, the amount of engineers that will interact with it over time increases significantly.

As the amount of people interacting with the code base and each other grows, the maximum amount of social relationships and interactions grows quite aggresively:

Where C stands for the total connections in the social network and n the amount of nodes in the network.

Large code projects don’t just break on code complexity, they break on failures of communication at scale.

We all know documentation is always a significant challenge when it comes to maintaining a large code base. Not necessarily because it’s hard but because incentives are not aligned.

Why would you write documentation if that is not rewarded from an organizational perspective? How do you justify spending time on documentation that you could be spending on building new features for the business?

Now what happens if we define our documentation in a code base in the form of prompts / knowledge in our codebase.

Let’s say for theoretical purposes we have a simple modular code base. Let’s say we have a few system wide prompts that engineers can use, as well as prompts that are specific to the modules and stored within them.

This does two things now.

1. A different engineer (perhaps a junior engineer) can read in plain english what the design and architecture of the codebase and modules is.

2. Engineers already familiar with the code base using LLMs for development can use these prompts when developing new features to get their LLM workflow to spit out the correct answer faster.

You see how documentation suddenly is more aligned with different incentives in the team?

We can now start to formalise in natural language what feature development in our code base ought to look like.

When I was a more junior engineer, I would learn a lot by observing the language used by more senior engineers. Whether it was:

• a technology (e.g snowflake)

• a development style (e.g test driven development)

• a software design pattern (low-coupling)

• an architecture (microservices)

Just being exposed to the language gave me the tools to ask better question to grow faster as an engineer. It gets engineers unfamiliar with the field much more quickly from an unkown unkown state to a known unknown state which is very powerful for learning.

Encoding these principles in well written prompts in the code base allows me to guide junior engineers faster through the learning process — perhaps we can even use this context to help them start a conversation with the LLM. For example one could create a prompt

Explain to me what it means for the code base to have <x attribute> and show me example in the current code base highlighting these principles.

Alright sure but how do we get a nexus of prompts into our code base you may ask?

CI/CD and prompts and code reviews

If we acknowledge that context is vital in a code base, this has to change our perspective to how we do code review and deploy new code.

Similarly to how one would require the inclusion of tests in a code base when developing a new feature, we should require a Pull Request to include the addition or modification of context and changes in the code base.

The best way to encode such a practise in CI/CD is still debatable but it’s possible to start small.

Whenever you’ve developed a new feature for a code base, simply have the LLM summarize your conversation, the learnings and what-not into crystallized rules and context for future conversations and commit these to version control.

Whenever a new pattern is added to the code base or a new module is introduced, a requirement in scoping SHOULD include the context provided to LLMs and humans in the code base.

Another thing to consider about prompts is having LLMs as code reviewers of pull requests.

We are now seeing the emergence of LLM driven code reviewing tools either as SaaS products or as part of CI/CD pipelines within organizations.

Let’s take a step back.

What makes a code review a good code review anyway?

We will all have different answers and preferences that are strongly influenced by the code base, the team, the organization, the industry and a whole lot more.

If I work in a strongly regulated industry like healthcare for example, I might want to have more critical review when it comes to compliance in accordance to say HIPAA/HITECH.

What else makes a code review good?

Getting the perspective from different kinds of developers with different styles, preferences, and expertise tend to improve the quality of the review.

With this in mind, we could create developer personas out of prompts and rules and we can use these to scale and enhance reviews from different kinds of engineers.

If for example, we have a security team in an organization that is stretched thin but still wants to verify and question specific components of all the pull request coming into an organization, that team can write the context and prompts for their developer persona and we would be able to integrate this perspective automatically in an AI.

Still, it might be a lot of work to write context and prompts from scratch which is why it would be a good idea to consider other ways to generate these contexts as well.

Creating organizational knowledge out of documents

Most large software organizations will have tons of design documents, feature specifications and requirements lying around that were used to start developing features in their code base. Why don’t we use these as input to create appropriate context in a code repository?

There’s actually a wide plethora of context produced in an organization in meetings, notes, docs, you name it.

The question becomes: are you integrating and using this as effectively as possible?

It’s possible for example to process documents and create vector databases out of them storing the knowledge as embeddings.

These embeddings can be the basis of an simple AI agentic workflow that take these inputs from across the organization and produce valuable context and documentation that can be used and version controlled in a code repository for future work and reference.

Okay, we’ve talked a bunch about collecting input for prompts but how should we actually model them in our code base?

Prompt modeling

One of the questions on prompts would be how to model them at scale in a code base / organization. While there are no good answers or frameworks yet, some of the question that came up for me are:

1. What sort of prompt are we storing?

   1. Is this a style guide?

   2. Is this external context? (e.g docs, syntax)

   3. Is this prompt part of the application?

   4. Is this prompt part of the development flow?

   5. Is this prompt meant for creative exploration?

   6. Is this prompt meant for critical feedback?

2. What is the metadata we want to store around our prompt?

   1. What model and settings did we use for this prompt?

   2. How long or short should our prompt be? (Cursor for example suggest < 500 lines for a rule)

   3. How “complex” is our prompt?

   4. How “vague” is our prompt?

   5. How do we label a prompt?

3. What overall structure do we maintain?

   1. Do we have a flat structure of composable rules or do we allow hierarchy?

   2. Do we store context in a central location, spread throughout the code base or do we have a separate repository altogether?

This is by no means an exhaustive list but in future posts and development in the field, it would definitely be interesting to think through the “meta” layer of this all.

Prisoners Dilemma in prompt sharing?

There’s an obvious catch to all of this prompt sharing. If a LLM-savvy engineer has a really effective personal system with highly effective customized prompts, custom context, etc, do they even want to be sharing their setup across an organization?

If one has a team with engineers that hold a zero-sum perspective that can create a classical prisoners dilemma

Let’s say we have an engineer A and B.

The organization decides to store and share prompts and context but both engineers have a personal choice as to how far they commit into this.

If we assume that both engineers have unique assets that can yield improved productivity and they do not talk honestly to each other, a funny situation arises.

If one engineer can have other engineers share all their knowledge but keep their unique assets to themselves, they could theoretically gain an asymmetrical advantage and create a wider margin in code output over other engineers output.

This changes drastically however if we measure ones impact on total output by the team.

In a more healthy team dynamic, what drives the impact of a senior+ engineer is not their individual output but their ability to lift and rise other engineers as a whole.

Making 10 other engineers 2x more productive makes you a 20X engineer, which is far better than being a 10X engineer all by yourself alone.

Now even if we have a healthy team dynamic, there is another big important question when it comes to sharing context: Who finally owns this all?

AI instructions: Fair use or intellectual property

If we consider prompt to actually be part of a code base what does this mean for the intellectual property in a a code base?

In software 1.0, intellectual property would be within the systems the organization as a whole has.

In software 2.0, the intellectual property might be the LLM and it’s weights (which is still a topic of much debate)

In software 3.0 however….english text is intellectual property of the organization?

Let’s try to extrapolate to a (possible) future where there is actually a ton of prompts and context defined for LLMs across code bases, either as part of the application or as part of the development experience.

Where do we draw the line of what is intellectual property of the organization and what is fair use outside of it?

If 20% of your code base is instructions for an AI, can’t we rule that it’s a crucial part of the application and therefore property of the firm?

Is the prompt valuable enough to even consider property or will this reduce in importance in future iterations of LLMs?

It’s a pretty funny twist in irony — AI tools scrape the internet and all of its intellectual property in it but then we claim intellectual property on it’s weights and the prompts.

"Your data team just got a Slack message at 3 AM: 'Why can the intern see customer SSNs?' Welcome to the wonderful world of RBAC sprawl."

Anyone working with large Snowflake data warehouses eventually hits the same wall: Role-Based Access Control doesn't scale with modern data complexity.

What starts as a simple "give Sarah access to the customer table" spirals into creating role_customer_readonly_q4_marketing_manager_aggregate_unless_emergency_then_full_access

If this sounds familiar, you've crashed into the limitations every data team faces when managing thousands of dbt models across multiple environments.

This article dives into the practical reality of implementing Attribute-Based Access Control (ABAC) in a modern data warehouse - specifically Snowflake + dbt. I'll walk through the design principles you need to consider when developing a system like this. What will become apparent is that although the tooling exists, implementing this properly remains a complex effort in today's technical landscape.

Why RBAC Breaks Down at Scale

Let's look at what's actually happening in your data warehouse right now.

Classical RBAC works fine when you have 10 tables and 5 users. It breaks down spectacularly when you're managing thousands of views and tables across multiple data marts, dynamic DAG-based transformations, column and row-level permissions that change based on user context, and business requirements that shift faster than your deployment pipeline.

With traditional RBAC, every permission request flows through your data platform team.

❓Want access to the new customer churn model?

📥 Submit a ticket.

❓ Need to see revenue data but only for your region?

📥 Another ticket.

🙈 Your data team becomes a permission factory full of permission monkeys.

It's not uncommon for my clients to have 10-20% of their data tickets related to access requests in some way, shape, or form. As the number of models and end-users consuming data grows, the potential access permutations grow exponentially.

Modern data transformation tools make this even more painful.

dbt operates on DAG-based selectors - you can easily target "all models tagged with 'marketing'" or "everything downstream of dim_customers."

But try doing that with traditional roles? Good luck.

ABAC changes this entirely. Instead of pre-defining every possible role combination, you define attributes and let the system dynamically determine access patterns.

The scenarios where ABAC becomes essential include multi-tenant SaaS platforms where different customers see only their data with admin overrides, healthcare HIPAA compliance where access varies by role, department, and patient relationship, and financial services with regulatory requirements that change based on user location and data sensitivity.

Building ABAC on Snowflake

Let's look at what implementing this actually involves.

Dynamic Masking and Column-Level Security

Before diving into tag-based implementation, you need to understand when to use row access policies versus column masking policies - they serve different purposes. Row access policies control which rows a user can see entirely, while column masking policies transform the data within visible columns at query time. Think of row access as "can this user see this customer's record at all?" versus column masking as "this user can see the customer record, but only a masked version of their credit card number."

Tag-based masking combines Snowflake's object tagging and masking policy features, allowing a masking policy to be set on a tag using an ALTER TAG command. When the data type in the masking policy signature matches the column's data type, tagged columns are automatically protected. This eliminates the need to manually apply masking policies to every single column containing sensitive data.

Here's how this works in practice:

Tag-based policies allow administrators to write a policy once, assign it to a tag once, and have the policy apply to many objects automatically. No more manually tracking down every email column across hundreds of tables.

Managing Policies and Sub-Policies

Each tag can support one masking policy per data type - if a tag already has a masking policy for VARCHAR, you cannot assign another VARCHAR masking policy to the same tag.

This constraint forces you to think strategically about your tag architecture. You might need separate tags like pii_email_internal and pii_email_external rather than trying to handle all email masking with a single tag.

Most teams underestimate the complexity required for production-ready masking. "Mask or no mask" thinking will leave you with security gaps and broken test environments. You need granular masking strategies that consider both security requirements and operational needs.

Every single Snowflake data type needs a policy - VARCHAR, NUMBER, DATE, TIMESTAMP, VARIANT. Miss one, and you have a security gap.

The Attribute Data Model

This is where things get more abstract.

Your attribute model determines everything else.

Binary access (masked/unmasked) is just the starting point.

You need to consider things like:

• Do we organize by business unit?

• By functional role?

• By data mart?

• By data lineage?

Here's another thing most teams miss when implementing this in the BI layer: you need a dedicated role per user for the BI layer. Each user connects through their individual role, and each role must be defined within at least one group.

State Management Between dbt and Snowflake

You're managing two sources of truth: your dbt graph with tags and configurations, and the Snowflake production environment with actual applied policies. Treat dbt as your "plan" and Snowflake as the environment that must match. This infrastructure-as-code mindset prevents configuration drift.

Then there is a painful truth in dbt.

DBT is very permissible when it comes to yaml structures, a bit too permissible.

❗️What happens if a model is not defined in a yaml?

❗️What happens if all columns aren’t defined?

❗️What happens if data is removed?

❗️What happens if data types change?

and so forth.

Your team must build automations that fills in default tags based on model location, naming conventions, or lineage position.

If you're going to automate tag management, use ruamel.yaml - it preserves comments and formatting when modifying dbt model YAML files programmatically.

Then there’s process questions.

Who actually maintains these policies day-to-day?

Who determines whether a user should be part of the group?

How often are groups evaluated?

Most teams underestimate this operational overhead.

Some more practical things around applying tags

For complete masking coverage, you cannot deploy models directly to production. Use staging tables with policy application, then atomic swaps. Any gap between model deployment and policy application means unmasked data exposure.

While Snowflake metadata operations are free, they're not infinitely scalable and putting unnecessary query volume on a virtual warehouse can create unnecessary strain. Instead of running separate ALTER statements for each column like so:

ALTER TABLE customer_data ALTER COLUMN first_name SET TAG pii_data = 'sensitive';
ALTER TABLE customer_data ALTER COLUMN last_name SET TAG pii_data = 'sensitive';
ALTER TABLE customer_data ALTER COLUMN email SET TAG pii_data = 'sensitive';
ALTER TABLE customer_data ALTER COLUMN phone SET TAG pii_data = 'sensitive';

Batch same-tag operations into a single transaction:

ALTER TABLE customer_data 
  ALTER COLUMN first_name SET TAG pii_data = 'sensitive',
  ALTER COLUMN last_name SET TAG pii_data = 'sensitive',
  ALTER COLUMN email SET TAG pii_data = 'sensitive',
  ALTER COLUMN phone SET TAG pii_data = 'sensitive';

Don't try to set multiple different tag types on the same column in one transaction - race conditions and inconsistent application await.

If you need to apply both pii_data and data_classification tags to the same column, do it in separate transactions to avoid conflicts.

Monitoring and Compliance

Snowflake's access history becomes your compliance lifeline. Y

ou need to track what data was accessed, in what format (masked/unmasked), by which user role, at what time, and for how long.

This isn't just nice-to-have monitoring - it's required for privacy compliance. When regulators ask "who accessed this patient data," you need answers, not shrugs.

You need real-time visibility into what's masked vs unmasked for each user, policy application success/failure rates, and unusual access patterns that might indicate policy violations.

This observability layer becomes critical for both security monitoring and compliance reporting.

Wrapping Up

Implementing ABAC isn't straightforward. If it were, everyone would be doing it already. You're trading the simplicity of "user has role X" for the complexity of "user with attributes A, B, C gets access level Y to data with attributes D, E, F."

But here's the thing - you're already dealing with that complexity. You're just managing it poorly with an explosion of specific roles and constant manual intervention. ABAC doesn't eliminate complexity; it organizes it in a way that scales.

Start small with clear use cases, invest in proper tooling and automation upfront, plan for the operational overhead of policy management, and build observability from day one.

Your future self (and your data governance team) will thank you when the next compliance audit rolls around and you can actually answer questions about who accessed what data, when, and why.

This concludes this week's deep dive into ABAC implementation.

Hope you picked up something useful for your own data governance challenges.

Got ABAC implementation questions or war stories? Hit reply - I read every email and love hearing about real-world data architecture challenges.

Need help implementing ABAC on your enterprise Snowflake + dbt setup? Consider our consulting services at

https://www.naronadata.com/ (formerly Zelytics).

Some helpful sources and references

• Snowflake Column-Level Security Documentation

• Snowflake Tag-Based Dynamic Data Masking

• Snowflake Access History and Query Monitoring

• ruamel.yaml Documentation

• Building Custom dbt Applications

Anyone that has been working with snowflake data warehouses with dbt will at multiple points throughout the lifetime of their warehouses engage into a cost optimisation effort of some sorts.

Data warehouses and associated costs around them are one of the largest non-human costs in a data team and as such are always a prime target for gaining most from a cost cutting effort.

It can sometimes feel a bit voodoo as to what exactly is the best way to manage costs. It’s not uncommon to devolve into platitudes or even to just assume that the compute layer will handle everything for you automatically.

That’s not to say the snowflake query engine does not do some impressive heavy lifting to optimizing cost/performance ratio on many queries but that does NOT mean you don’t need to put in more effort.

In this weeks newsletter I’ll go over some of the fallacies around snowflake cost monitoring and more advanced methods I have been using with my enterprise clients to successfully cut their snowflake spend by 50% on large 6 and 7 figure budgets when they’ve hit a wall in their cost optimization efforts.

This newsletter will assume you know the basics around snowflake costs and basic warehouse management in snowflake.

❌ Fallacy 1: Tracking queries through Role Based Access Control is enough

There’s many ways to divide queries into categories in your warehouse. You might split queries based on roles, based on dbt targets or even based on users. While this is a good initial approach, in order to get really detailed insights we will need to add more specific metadata because, as we will learn later, not all statements are created equally.

When deploying a data stack using snowflake, all sources, whether it be DBT, orchestrators, BI-tools or other applications will connect through some of the supported snowflake connectors.

Each connector supports the ability to add query tags, this can be a simple:

ALTER SESSION SET QUERY_TAG = '<tag_here>';

There are some packages that can help do this for a DBT project, most notably the open source package dbt-snowflake-monitoring which sets tags for all models and tests and provides very intuitive and comprehensive cost models and monitors for all queries running on the warehouse.

❌ Fallacy 2: Performance of all query types scales linearly with larger virtual warehouses / compute

Consider two types of queries. One is a simple but frequent COPY statement which copies data from an external S3 stage into a snowflake table. Another query is a complex CTE with several complex joins creating a table. These two statements do not have the same performance implications.

A COPY statement is more strongly influenced by the amount of files and the size of the files we’re trying to copy over. This means that, with more compute, we might find a similar performance but we get a higher cost.

For the CTE, we might see that running a 2x more expensive warehouse cuts query time by 60%. If our query runs long enough, this means we safe more money in run time compared to what we are spending on increased compute.

❌ Fallacy 3: Using a single virtual warehouse to support different kinds of query loads is an efficient configuration.

As an extension from fallacy 2, many teams will have their use of a virtual warehouse evolve quite substantially over time but this is then not reflected in the composition of virtual warehouses and the queries they carry.

For data engineering oriented query loads such as COPY statements or INSERTS, DELETES and UPDATES in source tables, it is easy enough to separate these out into their own virtual warehouse.

For DBT projects, it’s also possible to set a different snowflake virtual warehouse in dbt on various levels such as the model level, the materialisation level, a mart level and many more.

If you have properly tagged queries, it is possible to analyse trends in query usage and set a virtual warehouse of the right size in a DAG oriented approach.

For example, for many dbt internal tests and commands such as source freshness, these queries might benefit from being run in a smaller warehouse.

Now you might counter: “but these queries have low run time so does this matter for total cost”? It does and it brings us to the next fallacy.

❌ Fallacy 4: Short running queries are not important for snowflake costs

This fallacy actually consists of a few sub-fallacies.

❌ Fallacy 4a: Virtual warehouses can handle any arbitrary amount of concurrency

While it is true that it’s possible to scale concurrency through the use of larger warehouses and using multi-cluster warehouses in enterprise accounts, this does not mean a virtual warehouse cannot choke on high query volume.

The default concurrency is 8 and goes up to 32 queries for a virtual warehouse cluster.

Now, consider the scenario where we are running thousands upon thousands of queries in a short span of time like what we might have if we’re running a DBT source freshness command.

Every transaction has a basic overhead to be executed. Running a high volume of small queries can cause higher than expected query waiting time and cost.

❌ Fallacy 4b: Small queries are handled consistently every time in every warehouse

I ran an experiment for a simple overwrite for slowly changing data for a set of tables between 1000 records and 50 million records in a snowflake XL warehouse with thousands of queries load and observed a peculiar pattern: The most expensive queries were not from the largest tables nor were they consistently expensive for the same query even though the data in and out remained the same.

The variation was staggering, the most expensive query of the same kind was in the single $ range and the least expensive was a 1000X cheaper.

While this may not sound like much, if you are replicating your tables 6 times a day and you have 500 tables where 1 query exceeds $1 you end up with a bill of $3000 a day for this replication.

Needless to say, moving this to a smaller warehouse resolved the cost of these queries tremendously, as we had less congestion on the older large warehouse and lower cost while running queries at a similar amount of time.

Fin

This concludes this weeks newsletter. Hope you picked up something new this week and if you did, show us your support by following the newsletter!

As data engineers, we may be familiar with various ways to remove certain constraints. We may process data in chunks, process data in parallel, process it asynchronously, slap a data orchestrator on it and maybe even go crazy with auto-scaling policies in kubernetes.

Today, I want to talk about a ELT/ETL pattern that is often overlooked but can be quite powerful and flexible when implented properly. This pattern, is streaming.

….wait but I know what streaming is, isn’t that about data latency and using fancy frameworks like Kafka to process my data? Yes, that is a form of streaming but not the form this article will dive deeper into.

In this article, we wil dive deeper into various kind of streaming data types in python and how we can configure them to stream data from various sources to various destinations.

What is a “stream” and what does it look like?

For the purposes of this article, we will maintain the following loose definition of a “stream”

✍🏻 A stream is a sequence of potentially unlimited chunks of data made available over time.

A stream can be thought of as items on a conveyor belt being processed one at a time rather than all at once.

So essentially, we want to have a data type that can do two things:

1. Define a sequence of events to process one at a time

2. Allow to read one chunk at a time

Oftentimes you will see this being defined as “lazy” computing and it’s very common across many big data frameworks such as dask and spark. “Lazy” here refers to the fact that the chunk of data is only loaded and processed when it is asked to do so rather than precomputing everything up front and then processing each element individually.

As a data engineer, you’ve definitely used these sorts of datatypes before.

Let’s say you’ve loaded a CSV file with pandas before with chunking turned on, well, pandas will use a streaming like object in the background to facilitate this!

well…to be fair, if we are talking stritcly canonical python, you may be aware that we are specifically talking about iterators .

Now you may say: “Hey but I already know those, i’m outa here”.

Yes, you may be aware of iterators but there’s a suprising amount of ways that you can leverage them to stream data from and to sources in ways you may not expect.

If you don’t know what iterators are, the following section is for you.

A way to iterate and iterate and iterate

Feel free to skip this section if you are familiar enough with iterators .

An iterable is any Python object capable of returning its elements one at a time, allowing it to be iterated over in a loop. Examples include lists, tuples, sets, dictionaries, and strings. An object is considered iterable if it implements the __iter__() method, which returns an iterator.

An iterator is an object that represents a stream of data. It returns data one element at a time when the __next__() method is called. Iterators keep track of their current position. An iterator is created by calling the __iter__() method on an iterable, which returns the iterator object itself.

The Abstract Base Class (ABC) iterator protocol defines two methods that an object must implement to be considered an iterator:

• __iter__() returns the iterator object itself.

• __next__() returns the next item in the sequence. If there are no more items, it raises StopIteration.

A generator is a special type of iterator created using a function that uses the yield statement to return data one item at a time. Generators provide a convenient way to implement iterators without needing to write a full class with __iter__() and __next__() methods.

A for loop in Python can be used to iterate over an iterable object, executing the loop body for each item.

The yield statement is used in a generator function to return data without terminating the function. The function state is saved, allowing the generator to resume where it left off when next() is called again.

By understanding these components, you can effectively use and create iterators and generators in Python to handle sequences of data efficiently.

Using iterators for streaming data from various sources

In this section we will go over a few examples of common sources you may want to load from and to as a data engineer.

Streaming data from an API with HTTPX.stream

To stream data from an API, you can use the httpx library. This is particularly useful for downloading large files or handling continuous data feeds.

Streaming Data to and from S3 with S3FS

For handling data in Amazon S3, the s3fs library allows you to stream data directly from S3 buckets without downloading the entire file.

Streaming Data to a Data Warehouse with Snowflake Connector

When streaming data to a data warehouse like Snowflake, I have a separate article going in depth in to how to use the snowflake connector to do so. Here I will illustrate a few ways in which we can stream data with the snowflake connector.

When retrieving data, we can retrieve an iterator of pandas dataframes for examples. With each iteration, the connector will download the next chunk of results data:

It’s also possible to stream data TO snowflake by supplying it with a file stream object like so:

Streaming Data with SFTP using Paramiko

The paramiko library allows you to stream data to and from an SFTP server, useful for transferring large files securely.

By using these libraries and methods, you can effectively implement streaming data interfaces to handle large datasets efficiently across various platforms and services.

JSON schemas can represent YAML schemas

The first thing to highlight is that, at least since the 2009 specification of yaml 1.2 , yaml is a strict superset of JSON. What does this mean in practise? This means that proper implementation of the YAML 1.2 protocol will yield a parser that can parse both JSON and YAML. In other words, YAML can represent anything that can be represented in JSON but more. Conversely, this means we can use JSON tools to represent a subset of YAML specifications.

For the validation of JSON files, the JSON schema specification allows to express expected schemas for a JSON file through another JSON file in JSON schema format. We can use this specification to define the schema for a json file representation of a yaml file. Consider the following example yaml file:

app:
  name: MyApp
  version: 1.0.0
  debug: true
  database:
    host: localhost
    port: 5432
    username: user
    password: pass
  features:
    - authentication
    - logging
    - analytics

The JSON equivalent of this file would look something like:

{
  "app": {
    "name": "MyApp",
    "version": "1.0.0",
    "debug": true,
    "database": {
      "host": "localhost",
      "port": 5432,
      "username": "user",
      "password": "pass"
    },
    "features": [
      "authentication",
      "logging",
      "analytics"
    ]
  }
}

The json schema that would represent the above schema looks as follows in json-schema format:

{
  "$schema": "<http://json-schema.org/draft-07/schema#>",
  "type": "object",
  "properties": {
    "app": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "version": {
          "type": "string",
          "pattern": "^[0-9]+\\\\.[0-9]+\\\\.[0-9]+$"
        },
        "debug": {
          "type": "boolean"
        },
        "database": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "integer",
              "minimum": 1,
              "maximum": 65535
            },
            "username": {
              "type": "string"
            },
            "password": {
              "type": "string"
            }
          },
          "required": ["host", "port", "username", "password"]
        },
        "features": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "required": ["name", "version", "debug", "database", "features"]
    }
  },
  "required": ["app"]
}

As you can probably tell, the json schema defines an expected schema for a json file. The json schema specification allows for many different kinds of validation mechanisms. In the above example, you can see that it’s possible to indicate which keys an object should at least have through required . You can also see that we can define validations for specific fields such as a minimum and maximum port number and a regular expression pattern for version.

Considering the above, we can use json schemas to represent validation rules for yaml files, but how can we easily integrate this in our interactive development environment? If you are using VScode, the YAML extension by RedHat offers a very simple solution.

Custom YAML validation with JSON schemas in VScode

The first step is to install the YAML extension redhat.vscode-yaml in VScode

Now, in your settings.json file for VScode, you can create a map between json schema and yaml files to validate with that json schema. So for example:

"yaml.schemas": {
	"<absolute path to schema>": [
		"/**/*app.y*ml",
	]
}

The above will tell us that the json schema custom_app_yaml_schema.json will be used to validate any YAML file on our system that ends with app.yaml or app.yml (though technically the regex would pick up something like app.ybml too but that’s besides the point).

Now, let’s take the above json schema and use it with the app.yml we’ve described before.

Now to make potential validation errors more obvious when I open the yaml file, I add another extension called usernamehw.errorlens which adds a more obvious highlight for errors:

Now, let’s say we purposefully create some alterations of this yaml file that don’t fit the schema. We now get real time feedback from the JSON schema on our invalid configuration!

Pretty cool hu?

This can be a very powerful tool in your IDE. In the rest of this article we will go a little bit more in depth into json schemas and how we can work with them to configure various kinds of validations.

JSON schema customizations

JSON schemas can have a bit of a learning curve when it comes to writing correct expressions. There have also been quite a few releases over the years when it comes to specifications. Please keep in mind what schema your tooling supports. As of today, the YAML extension support schemas of format draft 7, so I will keep to functionality specific to that draft.

json-schema documentation is pretty all encompassing when it comes to capabilities. In this section I want to highlight a few parts of the specification that have been useful for me so far:

The root elements

Each json schema document will have a few minimum root elements:

• title : The title of the document

• $schema : A url to the json schema specification.

The $schema element here points to any of the official json draft specifications. We have actually seen one in the example schema before: https://json-schema.org/draft-07/schema. Go ahead, click that link. You will notice that the schema is defined in a json-schema like format!

Json schema types

The basic non-hierarchical data types are string, integer, number, boolean and null. These are types that cannot have nested or recursive interpretations. For those use cases, there’s arrays (equivalent to a python list or tuple) and objects (equivalent to a python dictionary). Let’s go into a little more detail of each.

Null and boolean

The simplest of these are null and boolean which have straightforward meanings for each programming language interface provided to json schema. For example, null will evaluate to None in python.

Numeric types

Integers and numbers have more validation available to them. It’s possible to define a number must be a multipleOf a certain factor. The example below will only accept multiples of ten:

{
  "type": "number",
  "multipleOf" : 10
}

Ranges of numbers are specified using a combination of the minimum and maximum keywords, (or exclusiveMinimum and exclusiveMaximum for expressing exclusive range).

If x is the value being validated, the following must hold true:

x ≥ minimum

x > exclusiveMinimum

x ≤ maximum

x < exclusiveMaximum

While you can specify both of minimum and exclusiveMinimum or both of maximum and exclusiveMaximum, it doesn't really make sense to do so.

The following example will allow values 0 through 99:

{
  "type": "number",
  "minimum": 0,
  "exclusiveMaximum": 100
}

String types

String types are the most flexible type. Strings have a variety of validation tools to their disposal.

The simplest validation tool is length .

The length of a string can be constrained using the minLength and maxLength keywords. For both keywords, the value must be a non-negative number.

The following defines that a string must be between 2 and 3 characters long:

{
  "type": "string",
  "minLength": 2,
  "maxLength": 3
}

Format can be used to define that a string has a certain format like an e-mail address. Do note that support for various formats depends on the implementation of the json schema at hand. The “canonical” supported formats as per specification can be found here.

For example, if we have an e-mail, we may define it as:

{
  "type": "string",
  "format": "email",
}

Finally, for the most flexibility we can user regular expressions through the pattern keyword. Keep in mind that json schema maintains a specific implementation of regular expressions which can be viewed here.

For example, if we want to define a string should start with { and end with } it can be defined as follows:

 {
   "type": "string",
   "pattern": "^\\\\{\\\\{(.|[\\\\r\\\\n])*\\\\}\\\\}$",
  }

Array

Arrays are the first type that we discuss that define more than one element.

There are two ways in which arrays are generally used in JSON:

• List validation: a sequence of arbitrary length where each item matches the same schema.

• Tuple validation: a sequence of fixed length where each item may have a different schema. In this usage, the index (or location) of each item is meaningful as to how the value is interpreted. (This usage is often given a whole separate type in some programming languages, such as Python's tuple).

There’s a few keywords that are important to understand for arrays.

Validating the type of all array elements with items and contains

Items can be used to define what the type of the elements in the list should be. For example, if we only want numbers in a list we can define that as:

{
  "type": "array",
  "items": {
    "type": "number"
  }
}

If we want at least one element to be a number instead, we can use contains to define that schema:

{
  "type": "array",
  "contains": {
    "type": "number"
  }
}

Tuple validation using PrefixItems

For tuple validation, we can use the keyword PrefixItems to determine the required schema of the tuple elements. Let’s say we have a tuple that defines an element street name and street type . For street name we just want to have a string. For street type we want to use a string from a specific set of strings. We can accomplish the latter through the enum keyword. This results into the following schema component:

{
  "type": "array",
  "prefixItems": [
    { "type": "string" },
    { "enum": ["Street", "Avenue", "Boulevard"] },
  ]
}

We can use the items keyword as mentioned in the previous section to define more specific constraints for elements. For example, Setting items to false disallows extra elements to be added in this list:

{
  "type": "array",
  "prefixItems": [

    { "enum": ["Street", "Avenue", "Boulevard"] },
    { "enum": ["NW", "NE", "SW", "SE"] }
  ],
  "items": false
}

Crucially, items accepts subschemas as well and with that, arrays are the first type we described in this article that can act as a “container” for other types:

{
  "type": "array",
  "prefixItems": [
    { "enum": ["Street", "Avenue", "Boulevard"] },
    { "enum": ["NW", "NE", "SW", "SE"] }
  ],
  "items": { "type": "string" }
}

There are more keywords that can be used with arrays but for the sake of brevity we won’t go over those here.

Objects

Objects are the mapping type in JSON. They map "keys" to "values". In JSON, the "keys" must always be strings. Each of these pairs is conventionally referred to as a "property".

The properties (key-value pairs) on an object are defined using the properties keyword. The value of properties is an object, where each key is the name of a property and each value is a schema used to validate that property. Any property that doesn't match any of the property names in the properties keyword is ignored by this keyword.

Let’s take the same example as the tuple validation example specified in the array section. We have a street_name and a street_type . In an object, we can define this as follows:

{
  "type": "object",
  "properties": {
    "street_name": { "type": "string" },
    "street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
  }
}

Objects have different keywords that encompass similar behavior to those found in arrays , however, One thing that objects are particularly well suited for is creating more complex schema compositions. We will go into more complex schema generation into the following sections because there is some context to be understood before diving into that.

Structuring a Complex Schema in JSON Schema: The URI model

When working with JSON Schema, it's advantageous to structure your schemas into reusable components, much like how you would structure your code with functions and modules. This approach not only promotes reusability but also makes your schemas easier to maintain and extend.

Schema Identification

Schemas are identified using non-relative URIs, which act as unique identifiers. Although schemas are not required to have an identifier, having one allows you to reference them from other schemas. In this context, schemas without identifiers are termed "anonymous schemas."

Base URI and $id

The base URI of a schema is crucial for resolving relative references. By default, the base URI is the retrieval URI of the schema. You can explicitly set the base URI using the $id keyword. For example:

{
  "$id": "<https://example.com/schemas/address>",
  "type": "object",
  "properties": {
    "street_address": { "type": "string" },
    "city": { "type": "string" },
    "state": { "type": "string" }
  },
  "required": ["street_address", "city", "state"]
}

In this schema, $id sets the base URI to https://example.com/schemas/address. Please do note that this does NOT neccecarily mean it is fetched from this webaddress.

Even though schemas are identified by URIs, those identifiers are not necessarily network-addressable. They are just identifiers. Generally, implementations don't make HTTP requests (https://) or read from the file system (file://) to fetch schemas. Instead, they provide a way to load schemas into an internal schema database. When a schema is referenced by it's URI identifier, the schema is retrieved from the internal schema database.

The URI https://example.com/schemas/address#/properties/street_address identifies the street_address property in the schema.

$ref for Reusing Schemas

The $ref keyword allows you to reuse existing schemas.

Example:

{
  "$id": "<https://example.com/schemas/customer>",
  "type": "object",
  "properties": {
    "first_name": { "type": "string" },
    "last_name": { "type": "string" },
    "shipping_address": { "$ref": "<https://example.com/schemas/address>" },
    "billing_address": { "$ref": "<https://example.com/schemas/address>" }
  },
  "required": ["first_name", "last_name", "shipping_address", "billing_address"]
}

Here, both shipping_address and billing_address reference the same address schema, ensuring consistency and reducing redundancy.

$defs for Local Subschemas

Use $defs to define subschemas that are intended for reuse within the same schema document. This keeps your schema organized and readable.

Example:

{
  "$id": "<https://example.com/schemas/customer>",
  "type": "object",
  "properties": {
    "first_name": { "$ref": "#/$defs/name" },
    "last_name": { "$ref": "#/$defs/name" },
    "shipping_address": { "$ref": "<https://example.com/schemas/address>" },
    "billing_address": { "$ref": "<https://example.com/schemas/address>" }
  },
  "required": ["first_name", "last_name", "shipping_address", "billing_address"],
  "$defs": {
    "name": { "type": "string" }
  }
}

Recursive Schemas

Schemas can be recursive, referring to themselves, which is useful for representing hierarchical structures like trees.

Example:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "children": {
      "type": "array",
      "items": { "$ref": "#" }
    }
  }
}

This schema defines a recursive structure where each children array can contain items that follow the same schema.

Bundling Schemas

For distribution, you might want to bundle multiple schemas into a single document. This can be done using $id in subschemas, creating a compound schema document.

Example:

{
  "$id": "<https://example.com/schemas/customer>",
  "$schema": "<https://json-schema.org/draft/2020-12/schema>",
  "type": "object",
  "properties": {
    "first_name": { "type": "string" },
    "last_name": { "type": "string" },
    "shipping_address": { "$ref": "#/$defs/address" },
    "billing_address": { "$ref": "#/$defs/address" }
  },
  "required": ["first_name", "last_name", "shipping_address", "billing_address"],
  "$defs": {
    "address": {
      "$id": "<https://example.com/schemas/address>",
      "type": "object",
      "properties": {
        "street_address": { "type": "string" },
        "city": { "type": "string" },
        "state": { "$ref": "#/definitions/state" }
      },
      "required": ["street_address", "city", "state"],
      "definitions": {
        "state": { "enum": ["CA", "NY", "... etc ..."] }
      }
    }
  }
}

Bundling schemas this way ensures all necessary components are included in a single document, making it easier to share and maintain.

Applying binary operators on schemas

The keywords used to combine schemas are:

• allOf: (AND) Must be valid against all of the subschemas

• anyOf: (OR) Must be valid against any of the subschemas

• oneOf: (XOR) Must be valid against exactly one of the subschemas

All of these keywords must be set to an array, where each item is a schema. Be careful with recursive schemas as they can exponentially increase processing times.

In addition, there is:

• not: (NOT) Must not be valid against the given schema

This can be used as follows for example:

{
  "allOf": [
    { "type": "string" },
    { "maxLength": 5 }
  ]
}

Wrapping up

This article went into depth on how custom validators can be configured for any set of YAML files with custom json schemas and the yaml extension on VScode. We went in depth into how JSON schemas can be constructed to validate various types of documents. In a next article, we will focus on applying this to a specific use case for data (analytics) engineering for maintaining YAML state of dbt yaml files by showing how we can interface with these validations through python as well.

When you’re working with snowflake as a data engineer, chances are you will be developing applications and capabilities of your data platform on top of snowflake.

At the very least, you will be interacting with third party tools that directly integrate with snowflake themselves such as data transformations tools like dbt, reporting tools like looker and data orchestrators such as airflow.

What underlies all of these is interacting with the snowflake platform through a programmatic manner. As such, we can view the development of snowflake based external applications as follows:

There is the snowflake platform with it’s capabilities which we can interface with in a few ways:

• Through an UI such as snowflake worksheets and snowflake snowsight

• Through the CLI interface snowSQL

• Through programmatic interfaces in various programming languages (to view all available connectors, check https://github.com/snowflakedb).

Almost every external application, SaaS or integration will use a programmatic interface or snowflake connector in some way shape or form to build out functionality.

An example of this would be the data transformation tool dbt. Dbt has a snowflake integration, but under the hood it will use the snowflake connector for python to manage executing SQL commands against snowflake.

Another example would be the snowflake plugin in VScode which uses the nodeJS connector to execute various SQL commands based on UI input.

Considering this, understanding the programmatic interface with snowflake allows to:

• Build custom data applications for your data platform

• Understand and debug (snowflake) third party tools

• Integrate Snowflake more natively with distributed computing on your data orchestrator

• Build out capabilities that are not quite handled by higher level APIs such as snowpark

• Understand the fundamental principles of how snowflake handles queries, results and more.

The components of a snowflake connector: The python example

In this article we’ll be exploring the python connector to understand snowflake connectors in general. As each connector implements the same fundamentals, we can use the example of python to understand the most fundamental principles of all connectors.

Fundamentally, there are two main components to control for in the connector:

• The connection

• The cursor

The connection configuration determines how you connect to snowflake from python.

The cursor is a database object used to retrieve, manipulate, and navigate through a snowflake result set. It’s used to send SQL instructions to snowflake as well as store results from any execute method performed.

The cursor has execution method that determine what SQL operation we actually execute against snowflake and how we process the result back into python.

Connection

This component will dictate how the python connector can connect to your snowflake account. The relevant python class is SnowflakeConnection, which will handle most of the configurations around connecting to snowflake and some more. There is a lot of things that we can set as of current release 3.10.1. Instead of going through every single setting, we will go through a few conceptual subcategories and go over the most important attributes you may or may not want to set. For a more extensive deep dive, feel free to refer to the source code of SnowflakeConnection.

Defining security and authentication parameters

I will start with security of snowflake connections as most settings around connectivity revolve in some way, shape or form around security. Considering the recent security breaches on the Snowflake platform of several large organisations, it is more relevant than ever to pay attention to this.

To connect to an account you will need the correct account identifier based on your authentication method. When connecting with a username, use the user attribute to define the snowflake username to log in with.

The most significant attribute is authenticator. This attribute will determine what authentication method is used to authenticate to snowflake. The table below shows the currently supported first-party authentication methods you’ll probably use most alongside some relevant details for each.

Username and password alone are vulnerable to various attacks such as phishing, brute force, and credential stuffing. Passwords can be weak, reused across multiple sites, and stolen in data breaches.

SSH key pairs are more secure than passwords because they use public key cryptography. The private key is kept secret and the public key is shared. However, the security relies heavily on the protection of the private key. If the private key is compromised or infrequently rotated, the security is significantly weakened.

Combining username and password with MFA significantly enhances security. Even if the password is compromised, the attacker would also need access to the second factor (such as a physical token, mobile app, or biometrics), making unauthorized access much more difficult. However, using a password still confers some of the weaknesses of general user+pass flows.

Single Sign-On (SSO) via Okta provides a higher level of security through centralized authentication. It often includes robust features like strong password policies, adaptive multi-factor authentication (MFA), and anomaly detection. The downside is that the security of the entire system depends on the security of the SSO provider. These providers are prime targets for hacks. Additionally, there may be additional cost involved with these solutions, often more suited to enterprise users.

Chances are, if you work in a company of any decent size, there will be some sort of proxy or VPN solution used to connect to things. For proxies, it’s possible to set things up through the attributes proxy_host, proxy_port, proxy_user and proxy_password. For VPNs, configuration will depend from VPN to VPN provider but usually, it’s possible to set a CA certificate for the VPN of choice. The snowflake connector uses pythons standard certificate store so a certificate can be simply set as:

cat public_cert.crt >> $(python -m certifi)
pip config set global.cert $(python -m certifi)

We may delve deeper in the future in topics around these security protocols, but these things should give you a rough idea how to implement these common setups.

One last thing to note about security is that there are various security options available such as insecure mode , ocsp_fail_open , port and more. The best advice here is to never alter these values. If you find yourself in the need to configure these sorts of settings to resolve issues around connectivity, it’s a lot more prudent to find the actual root cause of your issue rather than downgrading the security of your application to make a connection.

Defining database parameters

Within the connection configuration, there are several attributes that map to database components that you would typically use when interacting with snowflake. Using database , schema , warehouse and role we can define the characteristics of our database connection. This will follow snowflake role based access control configurations in your account. If you want the snowflake connector to check if your combination of database parameters are valid for your snowflake account, you can set validate_default_parameters to true which will give you an error if you try to connect with incorrect database parameters.

Defining how API data is parsed in python

When interacting with snowflake, at the core we’re exchanging data over the web which means we will have some sort of serialization and deserialization happening between the client and server. As such, there is a lot of “translation” happening between Snowflake API compatible formats and objects that are native to python. There’s a few important parameters to understand here, though you may not need to alter them often.

The python snowflake connector will handle database exceptions with a custom exception class that inherits from python exceptions. Essentially, this exception class will parse database errors from the API responses and convert them to python exceptions such that your python application can have native exceptions on database errors. If you wish to change the parsing, formatting or other handling of errors, you can write your own classes and overwrite with the errorhandler attribute.

When it comes to converting snowflake API types to python types, this is handled by a SnowflakeConverter class. Simply put, these classes determine how to translate types from API format to python. Generally this is handled pretty well, but, in some cases it may be useful to overwrite this behavior when you are developing your data platform if you are using certain libraries and data constructs that are not part of the standard library. You may for example want to add additional type validation through pydantic. To overwrite the converter class, you can use converter_class in your connection config.

The python snowflake connector can result many different data types. We will dive a bit deeper into different results data types later but suffice to say that we can influence how are results are returned to us through parameters in the connection class. Two settings that come to mind are json_result_force_utf8_decoding which will force the connector to decode all json strings as utf8 when returning json and client_prefetch_threads which impacts how many threads are used to fetch the first set of results. Again, we will get back to these in depth in a bit, but just be aware that some result parsing settings are configured at the connection level.

Defining session and timeout parameters

When connecting to snowflake to execute some SQL statements, we can view the time from start to end of the connection as a session. As such, it’s prudent to consider how we bound these sessions to prevent issues like hanging connections.

Most relevant parameters around this concept are timeouts. In the snowflake connection configuration we can set a few different kinds of timeouts:

When we hit any of the above timeouts, we can set a backoff_policy to determine how long the connector will back off between connection attempts. The snowflake connector implements linear, exponential and mixed backoff functions, though it is possible to define your own. The default is mixed_backoff(), this is to prevent the issue of the thundering herd problem where many independent processes may try to perform API calls simultaneously, straining the available API bandwidth.

💡 Warehouse level timeouts impact the snowflake connector too so be mindful of your overal timeout structure.

Finally, if we want to be able to distinguish in our snowflake logs which application was connected to snowflake through the API, we can use the application attr to add a name of the application (default = "PythonConnector"). This can be useful if we have multiple types of application connecting to snowflake and we want to keep track of connections made through the information schema login table.

Cursor

Once a connection object is established we can derive a cursor object from it through connection.cursor(). A SnowflakeCursor will handle execution of various SQL commands against a snowflake database, as well as manage the retrieval and storage of the results of those commands.

As some configurations happen as attributes of the cursor object, some as method choice for execution and some as attributes of execution methods, it makes more sense to consider cursor configuration from the perspective of capabilities.

In the following sections we will consider how a SnowflakeCursor can be used to read/write data, execute queries synchronously or asynchronously and how we can handle different formats to process queries in a data orchestrator friendly distributive fashion.

Synchronous and Asynchronous SQL Statements

There’s generally two ways to handle SQL statement execution with the snowflake API:

1. Synchronous: The application will send the query for execution and wait for the results to be available before continuing the program. The results are therefore a python object representation of the results.

2. Asynchronous: The application will send the query for execution and NOT wait for the results. The result is thus a query_id which can be evaluated later to retrieve results.

Generally, you’d want to use synchronous queries as they are easier to handle and cover most use-cases. To execute synchronously, simply run like:

conn.cursor().execute("CREATE WAREHOUSE IF NOT EXISTS tiny_warehouse_mg")
conn.cursor().execute("CREATE DATABASE IF NOT EXISTS testdb_mg")
conn.cursor().execute("USE DATABASE testdb_mg")
conn.cursor().execute("CREATE SCHEMA IF NOT EXISTS testschema_mg")

when fetching results for async queries, this requires a little bit more work, your program should determine whether a query is still running and whether it has been successful if finished. Then you can proceed to retrieve results as usual.

This might look something like:

from snowflake.connector.constants import QueryStatus

# ... process sending initial query...
query_id = conn.cursor.execute_async(cursor_execute_config).sqfid
# ... process receiving query results...
while conn.is_still_running(
    conn.get_query_status_throw_if_error(query_id)
):
    print(conn.get_query_status_throw_if_error(query_id))
    time.sleep(check_interval)
if conn.get_query_status_throw_if_error(query_id) == QueryStatus.SUCCESS:
    resp = cur.get_results_from_sfqid(sfqid=query_id)
    # To work with the results, we need to explicitly fetch them first.
    cur.fetch_arrow_batches()
    print(
        f"Async query completed successfully and cursor ready for fetching. {resp}"
    )
    return cur
else:
    raise ValueError(
        f"Query failed, status is {conn.get_query_status_throw_if_error(query_id)}"
    )

Note that we are checking the query status consistently until we reach a completed query with either an error or success message. Notice that running fetch_arrow_batches() will store results and their metadata pointers in the cursor object itself.

Fetching data with the snowflake cursor

Before we get into the separate fetching methods for snowflake cursors, we’re going to take a deeper look into how the python connector fetches data from snowflake.

How snowflake fetches large datasets

When working with Snowflake, it's important to understand the mechanisms behind data fetching, especially for large datasets. Depending on the configuration, the API employs different processes to fetch data:

• Small Datasets: The API can prefetch the entire response and load it directly into a Python object in memory. This is efficient and quick, making it ideal for smaller datasets.

• Large Datasets: For larger datasets, the API can be configured to use a chunking method. Initially, a chunk of data is prefetched into memory, similar to the process for smaller datasets. Subsequently, the API fetches metadata pointers to the next chunks of data. This process is structured like a chain, with each chunk containing a pointer to the next chunk's location.

Note that the prefetch response and chunk responses are returned from different URLs. The prefetch response is returned directly by the SQL endpoint the connector interacts with https://<account_identifier*>*.snowflakecomputing.com/api . The chunking responses will return pre-signed S3 urls with a default lease of 6 hours.

What happens under the hood is that snowflake has S3 resources specifically dedicated to your account/service user that you use to set up initial S3 connections with. Snowflake will provide you with chunks/partitions of your data through S3 through S3-presigned-urls using your service account id. These links are of a shape similar to:

https://sfc-<server>-<data-center>-customer-stage.s3.<aws-cloud-region>/<service account id>/results/<statement_id>/main/data_<chunk_id>?<aws-s3-presigned-url-details>

Please do note that from a security perspective, these links also need to be permitted by any VPN/proxy you might be using and you must be very careful about logging these links as ANY outside party can use these links to access the chunk data within the lease time.

The following is a visual representation of what happens under the hood:

Essentially, when snowflake processes your query, it will store the results into chunks in S3 and the iterator objects within the snowflake connection objects will essentially just download each chunk through a pre-signed s3 url into an object in memory of your choice.

Internal python data types for cursor results

Single process fetching methods

With this type of method of form cursor.fetch_<type>() , the snowflake cursor is able to fetch data from a synchronous or asynchronous query into several formats into a single process. Depending on the size of the data and program complexity there are several possible methods.

Fetchall

cur.fetchall() is the most straightforward fetching method. When a statement result is fetched with this function, a list of tuples is generated where each tuple element corresponds to a column in the result set. For example:

# command
cmd = """
select *
From  (values ('Dummy1')
            ,('Dummy2')
    ) A(Dummies)
"""

cursor.execute(cmd)
results = cursor.fetchall()

# results
results: [
        (
            'Dummy1',
        ),
        (
            'Dummy2',
        ),
    ] (list) len=2

Be careful however, this method will load all of the data directly into memory. This method is NOT advised to run for queries where large result sets are expected. This method can be quite useful when running sql statements that can have variable outcomes such as Data Control Language (DCL) statements.

Fetchone

cur.fetchone() is similar to the fetchall() method, but instead of fetching all the data into a list of tuples at once, it will fetch one record at a time. Running the same statement as above, we get:

results: (
    'Dummy1',
) (tuple) len=1

Fetchmany

cur.fetchmany() is similar to cur.fetchone() , but it returns nrows of data from the query result set. This can be useful for testing smaller subsets of the data before proceeding with the application code.

Fetch_arrow_all

cur.fetch_arrow_all() will prefetch the first chunk(s) of results into the cursor and provide a generator for fetching the metadata references to the proceeding arrow chunks. Again, do note that a fetch function using all will directly load everything into memory:

total_rows = 10000
cmd = """
select *
From  (values
"""
for i in range(total_rows):
    cmd += f"('Dummy{i}')"
    if i < total_rows-1:
        cmd += ",\\n"
    else:
        cmd += "\\n"
cmd += ") A(Dummies)"

cursor.execute(cmd)
results = cursor.fetch_arrow_all()

# results
results: (
    pyarrow.Table
    DUMMIES: string not null
    ----
    DUMMIES:
    [["Dummy0","Dummy1","Dummy2","Dummy3","Dummy4",...,"Dummy9995","Dummy9996","Dummy9997","Dummy9998","Dummy9999"]]
) (Table) len=10000

This method is useful if your application can handle arrow format well. The arrow object is also pretty memory efficient, the above result object is ±14 kb large.

FetchArrowBatches

Chances are, you sometimes have to query datasets that will be many times larger than the example above. If these datasets exceed the size of your memory allocation to the process, this will not suffice. For these situations, there is a set of fetching methods that use batches to retrieve data from a snowflake query.

let’s say we generate a much larger dataset of 10.000 rows by 100 columns

  # Define the number of rows and columns
  total_rows = 10000
  total_columns = 100

  # Start building the SQL command
  cmd = "SELECT * FROM (VALUES\\n"

  # Iterate to create rows
  for i in range(total_rows):
      # Generate values for each column
      values = ", ".join(f"'Dummy{i}_{j}'" for j in range(total_columns))
      cmd += f"({values})"
      if i < total_rows - 1:
          cmd += ",\\n"
      else:
          cmd += "\\n"

  # Finish the SQL command
  cmd += ") A(" + ", ".join(f"Col{j}" for j in range(total_columns)) + ")"

when we now fetch with cur.fetch_arrow_batches() this will produce a python generator which will produce 6 chunks of arrow tables:

cursor.execute(cmd)
results = cursor.fetch_arrow_batches()

# results
counter=0
for batch in results:
    counter+=1
    print(counter)
# results
1
2
3
4
5
6

where each chunk will match the format of the arrow_fetch_all tables.

An important thing to note is that these generators cannot be easily passed around to other processes or tasks which has implications for data orchestration tooling. While it is possible to compute each chunk through a for loop and submit it to separate tasks in your orchestrator, this is rather inefficient because you are passing around large amounts of data into inputs of your downstream tasks. For that purpose, there are ResultBatches which are described in a later section.

FetchPandasAll and FetchPandasBatches

The snowflake connector also support fetching and converting data into pandas dataframes. Running the same code as above, but for a table of 5x5, we get the following format of our results:

    results: <DataFrame({
        'COL0': <Series({
            0: 'Dummy0_0',
            1: 'Dummy1_0',
            2: 'Dummy2_0',
            3: 'Dummy3_0',
            4: 'Dummy4_0',
        })>,
        'COL1': <Series({
            0: 'Dummy0_1',
            1: 'Dummy1_1',
            2: 'Dummy2_1',
            3: 'Dummy3_1',
            4: 'Dummy4_1',
        })>,
        'COL2': <Series({
            0: 'Dummy0_2',
            1: 'Dummy1_2',
            2: 'Dummy2_2',
            3: 'Dummy3_2',
            4: 'Dummy4_2',
        })>,
        'COL3': <Series({
            0: 'Dummy0_3',
            1: 'Dummy1_3',
            2: 'Dummy2_3',
            3: 'Dummy3_3',
            4: 'Dummy4_3',
        })>,
        'COL4': <Series({
            0: 'Dummy0_4',
            1: 'Dummy1_4',
            2: 'Dummy2_4',
            3: 'Dummy3_4',
            4: 'Dummy4_4',
        })>,
    })> (DataFrame) len=5

The benefit of pandas dataframes is obvious to any data professional that has been working with data and python: it is much easier to do transformations. Pandas has an extensive API for doing so. The downside is a larger memory footprint, but. Similarly to fetch_arrow_batches(), we can perform fetch_pandas_batches() to produce a generator of pandas dataframes to work with.

Multi-process fetching: ResultBatches

A common situation when handling large amounts of data from a database is distributing the transformation workload across several worker nodes in a data orchestrator like airflow or big data frameworks like Spark.

The earlier fetching methods work fine when working with a single process but they don’t cover this use case very well.

When we want to integrate with systems like these, snowflake provides a convenient result format called a ResultBatch which we can access through the method fetch_result_batches(). Unlike our generators which are difficult to pass around in frameworks heavily utilizing python futures like prefect or there’s limited bandwith, these chunks can be serialized with a pickle.dump into a small efficient binary representation (think deca kilobytes). This binary sequence can be passed around in most data orchestrators just fine.

When this data structure is deserialized in a target process, it can be deserialized into any of the other representations.

Loading data into snowflake through python connector

There is many ways to load data into snowflake but for the purposes of this article I will only briefly describe the three main methods when using the snowflake connector for python (I can follow up in a subsequent article):

1. It’s possible to do a standard cycle of loading data into S3, connecting this to a snowflake stage and then issuing copy into command through the cursor.

2. execute_cursor() can be extended with a parameter filestream to push a python IO object directly to a snowflake table. This is useful for larger-than-memory, non-dataframe objects.

3. The snowflake connector library also has a submodule called pandas_tools which contains a function called write_pandas(). This can used to write a pandas dataframe directly to snowflake. It auto-generates and runs statements you would typically run to implement the first way.

Wrapping up

Navigating the Snowflake ecosystem as a data engineer opens up a world of possibilities for building robust data applications and integrating powerful third-party tools. At the heart of this ecosystem is the Snowflake Python connector, a versatile tool that enables secure, efficient, and flexible interaction with Snowflake's data platform.

If you found this article helpful or insightful and/or are interested in more, please sign up to this substack below!

Subscribe now

This is part 2 of a series on data code modelling for data engineering. In part 1, we went over abstract base classes in python and how these can be used to design software contracts between python classes:

In this article, we will go over python typing. A good type system design is the foundation of a robust code base. Python is extensively typed and has extensive support for generating custom types and using these types at various locations throughout the code through the Typing library.

💡 Python typing is an EXTENSIVE topic. Therefore, this article will only cover some essentials on this topic.

Why care about python types

There are a few main reasons why it pays dividends to lay out your python typing more carefully:

1. Types are an efficient expression of function. It is a lot easier to read and write code with proper typing annotation than dealing with tons of documentation.

2. Types can be used by static code analysers like mypy to provide insights on potential code issues BEFORE it is run.

3. Types can be used by python libraries to enforce specific class behavior. For example, dataclasses and pydantic can be used to create a data model class which uses types to enforce specific outputs (more on that in part 3).

Python type hints

A type hint is a annotation of sorts that can be added to python code to signify the types of various types of objects and attributes. Type hinting can be done at a few important locations in your code:

• Class attributes

• Function arguments

• Function outputs

• Variables

To illustrate this, view the example below:

You will notice we are using the type constructs str, int and the custom class DataFrame at several places in our code. This shows a powerful feature of typing: All python data types and user generated custom objects can be used as a type hint.

This means that it’s also possible to use constructs from third party libraries such as pandas or numpys as a type hint. Thus, a type hint like df: pd.DataFrame is completely acceptable!

Why does this work? Python type hints do NOT impact your python code during runtime. They can be used by other libraries to create additional behavior, but in the “vanilla” implementation, they don’t impact how your code runs which is intended by design.

Generic types

A subset of python types are called the generic types. A generic type is characterised by a syntax like type[…] where … can be any other type. This is generally reserved for types that can be constructed as supersets of other types. For example, let’s say we have a list of integers in python, this could be type hinted like list[int] . This is pretty powerful as it allows us to have a more precise definition of this these sorts of types in our code. This can get infinitely complex too so list[list[list[str]]] is also an acceptable type hint.

Usually it’s rather obvious a type is a generic type. For example, for collections like list, dict or set it makes sense they are generics. As of current writing, it is not yet possible to determine if a type is generic simply by checking the base type (like list) but it is possible to confirm a generic once it’s parameterised (e.g list[int] ). Generally this can be confirmed like so:

This is all just to show that, although generics are widely used, they don’t generally inherit from a single hierarchy, they’re rather objects with related implementations.

There are a bunch of generic types defined in various python standard libraries that help a user implement and understand these types correctly. A non exhaustive list is provided below:

Typing

• tuple

• list

• dict

• set

Collections

• collections.deque

• collections.defaultdict

• collections.OrderedDict

• collections.Counter

• collections.ChainMap

• collections.abc.Awaitable

• collections.abc.Coroutine

• collections.abc.AsyncIterable

• collections.abc.AsyncIterator

• collections.abc.AsyncGenerator

• collections.abc.Iterable

• collections.abc.Iterator

• collections.abc.Generator

• collections.abc.Reversible

• collections.abc.Container

• collections.abc.Collection

• collections.abc.Callable

• collections.abc.Set

• collections.abc.MutableSet

• collections.abc.Mapping

• collections.abc.MutableMapping

• collections.abc.Sequence

• collections.abc.MutableSequence

• collections.abc.ByteString

• collections.abc.MappingView

• collections.abc.KeysView

• collections.abc.ItemsView

Contextlib

• contextlib.AbstractContextManager

• contextlib.AbstractAsyncContextManager

Queue

• queue.LifoQueue

• queue.Queue

• queue.PriorityQueue

• queue.SimpleQueue

Re

• re.Pattern

• re.Match

Others

• dataclasses.Field

• functools.cached_property

• functools.partialmethod

• os.PathLike

• shelve.BsdDbShelf

• shelve.DbfilenameShelf

• shelve.Shelf

• types.MappingProxyType

• weakref.WeakKeyDictionary

• weakref.WeakMethod

• weakref.WeakSet

• weakref.WeakValueDictionary

Annotated types

An annotated type can be seen as a custom type definition in python. It’s possible to define a type and associate some metadata with it. Generally, the metadata from these annotated types can be used to add additional instructions for validation of data. We will get back to leveraging that when we get into pydantic , but for now, just know you can define an annotated type like so:

Functions or other callable objects can be annotated using collections.abc.Callable or typing.Callable. For example, Callable[[int], str] indicates a function that takes an int and returns a str. An example is:

The subscription syntax for Callable requires two values: the argument list and the return type. Using an ellipsis ... as the argument list indicates that any parameter list is acceptable:

Special types and forms

Self

Self can be used to reference to the instance of a class object the code is currently operating on. We’ve seen this a lot in part 1 of this series when working with classes, but an example is:

This is useful for building out functionality between different components of your class.

Any

The Any type simply indicates a object can be of any type. As such it does not convey any additional information other than indicating to static code checkers this variable can be any type.

Use this construct sparingly as using this a lot usually indicates you are not clear on the functioning of your code. If you are in the need of defining a generic function working on various types, use a TypeVar instead:

Generally, typevar is better because:

1. Type Safety: TypeVar helps maintain type consistency, which means the function or class can work with any type while ensuring that the same type is used throughout. This prevents type-related errors that could occur if different types are inadvertently mixed.

2. Code Readability and Intent: Using TypeVar clearly communicates the intent that a function or class is designed to be generic and work with any type, but consistently. This makes the code easier to understand and maintain.

3. Static Type Checking: Type checkers (like mypy) can provide better error checking and autocompletion support when TypeVar is used. This can catch potential bugs at development time instead of runtime.

Union

Union is a typing construct that indicates the object it is encapsulating can be either type.

For example writing Union[str,int] defines a object as being either a string or an integer. Please do note this also means that when you define Union on a generic type which can be paramterized with another type like dict[Union[str,int]] , this will accept dictionaries that contain all strings, all ints or a combination of the two.

Please be aware of this distinction: Union does NOT cover the case where we are using strings and integers mixed together. Rather, it’s an either or.

For dictionaries in specific, it is worth exploring the TypedDict which allows you to define more explicit typing on specific types of keys van values.

Generally however, if you need to be this elaborate with typing, it usually makes sense to define a dataclass or pydantic model instead (more on that in part 3).

Optional

As the name implies, optional indicates a object is optional. This is most often seen with optional class attributes that are only used within specific contexts.

Literal

The literal type can be used to define a literal instance of a type. For example, type hinting with Literal[42] will indicate this object should always have a intenger value of 42 . This is useful for configuration-like objects that have a set of hard coded values.

If there is a larger amount of objects that depend on this configuration, it can be beneficial to define it as an enum instead.

Inspecting types in classes

Often when you are not too familiar with the inner workings objects in a code base or external library it can be difficult sometimes to determine the right type to use when laying out your typing system. In the past, I would have resorted to reading a bunch of source code to figure this out. It is also possible to determine this during run time by inspecting classes and variables.

Using vars

The vars function returns the __dict__ attribute of the given object, which contains all the attributes of the object.

Using dir

The dir function returns a list of all the attributes and methods of the given object.

Using type

The type function returns the type of the given object.

Using the inspect Library

The inspect library provides several useful functions to get information about live objects, such as modules, classes, methods, functions, tracebacks, and code objects.

These methods and functions can help you inspect and understand the types used in classes and objects, making it easier to implement type hinting and understand the structure of unfamiliar codebases.

With that, we’re going to conclude this section on python typing. Do note again that this is a massive topic in python and we’ve just scratched the surface here.

Moving on, you may wonder (or know) that typing and type hints in and of themselves don’t enforce anything in your code. This is the natural design of python as a dynamically typed language.

However, how can we take these typing constructs and actually do something in our code other than document things? This is where a framework like pydantic comes into play. In the following section, we will describe how to use pydantic to create data models in your code that leverage our previous knowledge on abstract base classes and typing to create a much more powerful data code model.

If you liked this and want to read more, please support the newsletter with a subscribe

Subscribe now

When working in data engineering, you will often have to work with python code bases to manage all sorts of processes in your data stack from ETL, to orchestration, to infrastructure management, to observability and much more.

One view of data engineering is that of a blend of software engineering and data science. Oftentimes as data engineers, we will spend time learning about general code best practises in software engineering and try to apply them to our domain.

However, a lot of the times these general principles fall short of the standards we would like to achieve.

On top of that, there has been a lot of development of python and the ecosystem around it so it’s possible to be a lot more intentional about our code design.

The big picture

Let’s take a step back and think about the bigger picture here. Python as a language is a strongly but a dynamically typed language. Simply put, python has a strict type system but how it used during runtime is flexible. If designed for then, python can get a lot closer (though not quite) to the type safety provided by statically strong typed languages.

Although the default configuration of python code is fine for development, for production systems we want a lot more guarantees about the consistency of our code. This is where a data code model comes into play. This is different for every project, but this article and follow up articles will provide a framework by which you can make your own informed decisions.

Creating a solid code data model leads to dramatically different trajectories for the amount of scope a team can handle.

When the focus is solely “ship fast - fix later”, this builds tech debt until a point of critical stress. At this point, the code base is bearing so much technical debt your teams time is absorbed with bug fixes, ad-hoc requests, refactoring and more. Development velocity reduces significantly until a point of legacy is reached. At this point, there is too many features, dependencies and dollars at stake to change trajectory so the company is left with a legacy code base which becomes increasingly slower to change.

When working with a proper code data model, initial velocity is lower. However, after some initial hurdles you reach a point of bliss after which developing more features and complexity becomes exponentially more easy until a higher level of peak complexity is reached.

Some of the reasons why is:

• Improved readability → Less documentation burden.

• Lower fragility of code changes → higher development velocity.

• Faster time to resolution on bugs → Better uptime.

• Static and runtime controls → Less need for extensive unit testing

• A higher capacity for complexity and features → More value, lower costs

So then, how to go at this concretely? In this article series, I will go over the data code model to structure your python code for data engineering. In this article, I will focus on the first component: Abstract Base Classes

Layer 1: Abstract Base Classes

Abstract Base Classes (ABCs) in Python serve as templates for other classes. They outline a set of definitions that must be created in any subclass that inherits from them. The primary purpose of ABCs is to ensure a consistent interface across different implementations. By defining an abstract base class, you're essentially setting a contract for what functionalities a set of classes should have.

An example of a abstract class is shown below

In this simple example, we have a abstract base class Animal, which prescribes any subclass needs to implement a make_sound and move method. We cannot instantiate the Animal class itself directly as it’s used for templating. We also cannot instantiate the subclasses if they don’t implement the required methods. So, abstract base classes are a way of enforcing a contract for a specific set of classes. That is, it defines a set of prescriptions of how a class should be constructed and typically resides in the first most basal layers in a code base.

Abstract Base Class Methods

The ABC library in python is very small at <200 lines of code as of today. As such there’s really only a few simple components from this library that will cover 95% of use cases. The @abstractmethod decorator is the main interface to create abstract definitions. It can be used in conjunction with a few types of methods in a class:

Abstract classmethod

A class method is a class function that can be used without instantiating a class object first. Creating a class object can be done by combining the @abstractmethod with a @classmethod decorator like so:

In this example we create an abstract base class for DataSource with functions configure and execute_query. The subclasses SQLDataSource and NoSQLDataSource are inheriting from DataSource and they will need to implement these methods now. Notice the difference between a classmethod and an instance method (“regular method”). The classmethod does not need an instance of the class, but has access to (default) attributes of the class through cls, whereas an instance method is more tightly bound to that instance of the class and can access the attributes of the current instance through self. A classmethod then, is for example useful when dealing with more complicated shared setup.

Abstract staticmethod

A staticmethod is essentially a regular python function that is tied to the namespace of a class. This method can be called without a class instance (unlike regular functions added to a class) and it does not have access to the internal properties of the class (so no access to self or cls). Similarly to an abstract classmethod, you can create these by combining the @abstractmethod with a @staticmethod decorator like so:

Static methods are great for utility classes that keep a set of functions for similar purposes or processes that can be used across functions and classes that are independent of each other.

Abstract property method

A property is a class function that is used to create a attribute for the class. Again, creating an abstract property can be done through combining @abstractmethod with a @property decorator like so:

Properties are useful when the definition of a class attribute is dependent on values of other attributes within the class or when this attribute requires a more elaborate calculation to set. Note that properties are separate entities from class attributes.

Using @abstractmethod with other function constructs

As you have probably noticed by now, all of the above methods follow a very similar structure where any function can be decorated with @abstractmethod to make it abstract. This is very flexible and allows for a lot of different types of flexible definitions. Again, check the source code, it is pretty straightforward. A few more advanced examples of types of functions that @abstractmethod can be applied to with useful purposes in data engineering are shown below.

Abstract Context Manager

Context managers in Python provide a convenient way to allocate and release resources precisely when needed. By using the with statement, context managers ensure that resources are automatically managed (such as opening and closing files, or acquiring and releasing locks) without needing explicit cleanup code. This is useful for a few different types of operations:

• Handling files

• Handling database connections

• Asynchronous and Concurrent operations

The contextlib library in python provides excellent construct to create your own context managers and combining this with @abstractmethod allows to create a data model for code covering these very common use cases.

For example, let’s say we want to define a set of file handling context managers that are specifically written for encrypting data. We want to make sure that if our encryption fails, we still close any file we opened. We may also want to have custom exception behavior. This may look something like this:

the contextlib library also provides constructs for defining async context managers which are invaluable when working with async code. This is esspecially relevant when dealing with many types of connections such as database connections or API connections:

Abstract Collections

The python standard library provides ABC class templates for specific types of classes in the collections library. These can be used as development guides when developing more complex code constructs such as iterators and asynchronous code. It is beyond the scope of this article to delve deeper into the implementation of these (potentially) performance enhancing implementations.

Concluding remarks

That’s it for abstract base classes! Although simple, abstract base classes can be an invaluable component for generating inheritance based object-oriented data models for your python code bases. However, abstract base classes are still not the complete picture.

Although we can define what our classes should implement, we are still in the dark when it comes to types. Also, what if we want to define more stringent implementation control without creating strong object dependencies? Expanding into typing will help us address some of these issues.

If you have found this helpful or if you want to stay up to date on follow up articles on the data code model or related topic, do not hesitate to subscribe below or share the post.

Resources

• Abstract Base Class Documentation https://docs.python.org/3/library/abc.html

• Abstract Base Class Implementation https://github.com/python/cpython/blob/3.12/Lib/abc.py

• Difference between class method, static method and regular method https://realpython.com/courses/python-method-types/#:~:text=Regular

• https://github.com/lord63/awesome-python-decorator has a good collection on python decorator resources

• https://www.integralist.co.uk/posts/python-generators/ ⇒ has a good collection on iterators, generators and coroutines.

• https://docs.python.org/3.7/library/collections.abc.html#collections.abc.Iterator ⇒ provides a list of built in abstract base classes

• abc collections source code ⇒ https://github.com/python/cpython/blob/3.12/Lib/_collections_abc.py

• contextlib ⇒ https://docs.python.org/3/library/contextlib.html#module-contextlib

What?

It helps to view any data initiative, small or large, as a small startup. In the startup world, one of the most important things to do is finding product-market fit.

This is all about understanding your customer and what they REALLY want.

In data, we can look at this in a similar way: How do we find out what efforts we should prioritize for our stakeholders?

In that spirit, here is a list of 30 questions to ask your stakeholders or yourself to figure out if the current efforts make sense:

1. What are your top objectives for this week, month, quarter, and year?

2. What is a typical task you perform often and would like to see more automated?

3. What are the top objectives of your stakeholders?

4. How do you measure success?

5. What data sources are you already using?

6. What data sources would you like to use more of?

7. How long would it take our team to build this?

8. What is the estimated lifetime value of this work?

9. Is there a depreciated return on value over time?

10. How many engineering hours will it take to maintain?

11. Can the responsibility easily be transferred, or is this dependent on 1-2 people?

12. What is the opportunity cost of doing this? Put differently, what other pursuits would need to get sacrificed?

13. Does it make more sense to buy a solution or build something in-house?

14. How will our team measure engagement of our new solutions with stakeholders?

15. In dollar terms, how much labor cost is this solution saving over time?

16. Does this solution drive more revenue? How do we prove this?

17. Is this solution reducing company liability? If so, how and how much?

18. Has this been done before? If so, how much time did it take them? If not, is there a good way to estimate how much time this would take to do?

19. How is this data going to be acted upon? If a stakeholder requests real-time but the actions are manual, consider what real-time really means here.

20. Apply Occam's razor to your solution: How can we achieve the results with the least amount of necessary permutations in our existing infrastructure.

21. Would this solution be very visible to the organization? If not, it may be harder to defend the value of this effort.

22. Is anyone else in the organization currently tackling this problem? Are we duplicating efforts? This requires talking to more people than just your direct stakeholders.

23. Has there been any effort in the past to address this issue? If so, get all the possible learnings you can get.

24. Are we considering this because it’s popular knowledge or a new shiny state-of-the-art tool, or is it truly relevant for our business?

25. How bad would it be if this effort fails?

26. What would be the repercussions of not doing this? How fast will the business feel the impact? How large would this impact be? Be concrete.

27. Can we deploy a quick minimum viable product to get buy-in earlier?

28. How confident am I we can deliver on our promises within a reasonable amount of time?

29. How many dependencies does this project have? Is this something we can run mostly internally, or does it need coordination with a lot of other teams?

30. Does this solution keep pace with the growth in scale of the business? How long will it take before this becomes obsolete?

What are some questions you like to ask stakeholders?

Like content like this? Support the newsletter with a subscribe!

Subscribe now

It all starts with local development environments

There’s several ways to drive impact in a tech driven organisation.

One of the ways to drive significant impact is improving the development workflow of other developers.

It’s a simple calculus: let’s say we save 5 hours of development time for a typical analytics engineer. If we have 10 AE’s, that amounts to 50 hours a week. Translated to salaries, this can amount to several thousands of dollars in generated value each week. The payout in productivity is exponential over time.

If VScode is the default code editor of most of your team, VScode devcontainers are a great way to go about this.

What is a VScode devcontainer?

A Visual Studio Code (VSCode) Development Container, or devcontainer, is a feature provided by VSCode that allows developers to create and configure a consistent, containerized development environment.

This feature utilizes Docker containers to encapsulate the development environment, ensuring that it includes all the necessary dependencies, tools, and settings required for a project.

Before we dive into create a local environment, let’s first consider some of the key concepts for configuration.

Settings.json

For any VScode environment, it is possible to configure the settings of the environment in settings.json . This json file can be used to set configurations of your VScode environment as well as all of the currently installed extensions. Try this for example:

(Mac) CMD+SHIFT+P > Preferences: Open User Settings (JSON)

(Windows) CTRL+SHIFT+P > Preferences: Open User Settings (JSON)

This will open the settings.json file for your user. Now try add the following to the json

"workbench.colorCustomizations": {
    "activityBar.background": "#ff00d4",
}

You should see your activity bar turn pink now. That’s sweet but we want to do more than that ofcourse.

Dockerfile and Docker compose

VScode devcontainers can be built with regular Dockerfiles. My suggestions would be to use any official base image on Dockerhub. As we will be setting up dbt, my suggestion would be to use a official python image. Make sure to consider the following things:

• Docker image size: The larger the docker image, the longer it will take to do the initial download and build for your environment. Python dockerhub has images tagged with slim which are generally 3X smaller. These are based on Debian.

• DBT python version compatibility: At the date of this article, this can be found in the dbt docs at https://docs.getdbt.com/faqs/Core/install-python-compatibility

• Opt for docker compose over running only dockerfile. Using docker compose allows us a greater deal of flexibility when it comes to setting up the docker environment around the dev container (i.e any custom co-containers, networks, etc..).

Devcontainer.json

The devcontainer.json is the main configuration file for the devcontainer. It can be viewed as a combination of docker orchestration commands and the settings.json file. You can view the documentation on this file here. To view the full json schema spec, check the devcontainers specification.

💡 Tip: As raw json schemas tend to be a bit difficult to read, try using https://github.com/coveooss/json-schema-for-humans to create a nicer representation for easy viewing.

I’ve published a more readable HTML document for the schema generated with json-schema-for-humans here. Feel free to clone the repo and open the HTML file on your local machine.

The specification are quite extensive, but it helps to think of the settings in a couple of broad categories:

• Devcontainer metadata ⇒ project name, where to find files, etc…

• Container specifications ⇒ Configuration for docker, docker compose, etc…

• Customizations ⇒ Any settings you want to pass on to the settings.json in the devcontainer.

• Features ⇒ Configure what functionality can be included within the dev container to be used.

What is VScode devcontainers doing in the background

The following general steps occur:

• First the container is built.

• Then a VScode server instance is inititated within the devcontainer. Thus, the implementation is akin to running VScode server on a remote machine.

• VScode server will install all extensions and configure any settings as provided in the configuration for the devcontainer.

• There’s some quality of life improvements. For example, whenever a browser is invoked to open on a remote without a desktop, usually this fails without any additional configuration. VScode however uses a custom configuration to actually open the browser on your host machine!

  • Keep in mind with this however, this can lead to unexpected behavior at first when working with things like browser based SSO. These applications will typically need additional configuration to work.

Creating the setup!

Now that we have covered the main concepts, let’s start applying them by creating a local dev environment with dbt.

Jaffle shop

For demo purposes, I will use the official dbt example project jaffle shop as a base project to showcase the devcontainer setup. You can check the jaffle shop demo together with the dbt devcontainer configuration in this public repo https://github.com/2tony2/dbt-devcontainer-demo/tree/main. Feel free to clone, fork and play around with it.

Installation requirements on your local machine

To make sure you’re running an as stable as possible deployment, ensure you have the following things installed / up to date:

• The Operating System of your machine

• Docker desktop

• The latest version of VScode

• Have at least the following VScode extensions installed:

  • ms-vscode-remote.remote-containers

  • ms-azuretools.vscode-docker

  • ms-vscode.remote-explorer

File setup

To create a devcontainer environment, make sure to clone a repo containing your dbt project (or the example project here if you’re following along).

Next, create a folder called .devcontainer at the root of your github repo. VScode will automatically pick up any devcontainer configurations from this folder.

Create the following empty files within the .devcontainer:

• Dockerfile

• docker-compose.yaml

• devcontainer.json

• devcontainer.env

• requirements.txt

• init (subfolder)

We will now go over the configuration for each individual file

Dockerfile

For the Dockerfile we have the following configuration

What this docker file do in general is install python, intall some OS dependencies, install python dependencies as defined in requirements.txt and store shell history so it can be passed from devcontainer to devcontainer.

Next, we set up the docker compose.

Docker compose yaml

A super simple docker compose file may look something like this:

This is a simple setup that will create a local-dbt-env service and configure it to run the linux/amd64 version of the base image by default. The other configurations ensure command history is retained between sessions and the command will ensure the container keeps running after being created by introducing an infinite loop. We can pass on any environmental variables through the env_file.

An example of a more advanced setup which uses SSO to connect to snowflake would be the following:

I will expand on setting up more advanced docker networking configurations in another future article.

After we have our simple docker environment, let’s set up the main configuration for the VScode devcontainer.

Devcontainer json

This is going to be the main setup for the dev container:

You can see we are installing the following extenions:

• innoverio.vscode-dbt-power-user ⇒ This plugin supercharges the local dbt development environment. This adds things like model lineage to your VScode UI.

• eamodio.gitlens ⇒ The free features include an added git blame to each line of code. This allows you to view what commit+author changed a line most recently.

• GitHub.copilot ⇒ An AI assistant that helps with code generation during development. Requires an active subscription on GitHub.

• dorzey.vscode-sqlfluff ⇒ A popular open source SQL formatter.

• oderwat.indent-rainbow ⇒ Adds colored indentation to code to aid in seeing indentation.

• redhat.vscode-yaml ⇒ allows us to do real time syntax highlighting of our yaml files based on json schemas.

• GitHub.github-vscode-theme ⇒ A color theme consistent with GitHub. Adds syntax highlighting similar to GitHub as well. Helps with consistency.

• mechatroner.rainbow-csv ⇒ Make every column in a CSV a distinct color. This helps with viewing seed files.

• usernamehw.errorlens ⇒ Add better visual indication of any formatting issues.

A few things to note about the configuration of this setup:

• We can adjust the amount of resources the docker container will be requesting through hostRequirements. Keep in mind that your docker desktop application should be configured to be able to provide these amounts of resources within the settings.

• The key vscode and then settings defines the settings.json for the dev container environment. Here’s a breakdown of some important settings

  • "extensions.verifySignature": false ⇒ Some VScode devcontainer environments will have very slow installation steps. (python and pylance can notable be slow). This setting helps to speed things up. Do NOT use this with extensions you do not trust.

  • "python.interpreter.infoVisibility": "always" ⇒ This tells VScode to always have a python interpreter available. DBT poweruser requires this as it is using constantly running python scripts to provide inputs to the UI elements.

  • "python.defaultInterpreterPath": "/usr/local/bin/python" & "dbt.dbtPythonPathOverride": "/usr/local/bin/python" ⇒ This will set the default python interpreter to the python interpreter where we install all of our requirements. This is needed for DBT poweruser to work well.

  • "files.associations": { "*.sql": "jinja-sql", "*.md": "jinja-md" }, ⇒ This tells VScode to interpret sql and md files as jinja-sql and jinja-md files. DBT poweruser parses files with jinja. Note that we leave out the yml files as is included in the default setup. This is so we can get other YAML functionality in here with the YAML extension.

  • "[yaml]": { "editor.defaultFormatter": "redhat.vscode-yaml", "editor.formatOnSave": true ⇒ these are settings for YAMl files. We select the redhad.vscode-yaml extension to interpret yamls as this allows syntax highlighting for yaml files.

  • "yaml.schemas": { "<https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json>": [ "/**/*.yml", "!profiles.yml", "!dbt_project.yml", "!packages.yml", "!selectors.yml", "!profile_template.yml" ], "<https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_project.json>": [ "dbt_project.yml" ], "<https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/selectors.json>": [ "selectors.yml" ], "<https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/packages.json>": [ "packages.yml" ] } ⇒ This will configure the yaml plugin to use the officially provided dbt-json schemas for various types of yaml configurations in your dbt project. To view these, check https://github.com/dbt-labs/dbt-jsonschema.

  • "initializeCommand": "echo 'Welcome to the DBT development environment!'" ⇒ There’s a bug in vscode Devcontainers that won’t start up devcontainers that do not have an init command. You can just add anything in here.

  • "postCreateCommand": "python /dev/init/init.py", We can create a custom post create command that is run after the docker container is created. This is helpful for initializing the environment.

  • "privileged": true ⇒ runs the container as root. Do NOT expose this container to the world wide web for your own safety reasons!