Apr 28

Solving correlated subquery error in BigQuery

Recently I had to create an array whilst selecting from another array in Google BigQuery: SELECT ARRAY( SELECT AS STRUCT b.Column FROM UNNEST(a.Parts) p LEFT OUTER JOIN TableB b ON b.JoinColumn = p.JoinColumn ) FROM TableA a Unfortunately trying to use an outer join prompts the error message: Error: Correlated…

Bigquery

1 min read

Solving correlated subquery error in BigQuery

Recently I had to create an array whilst selecting from another array in Google BigQuery:

SELECT ARRAY(
  SELECT AS STRUCT
    b.Column
  FROM UNNEST(a.Parts) p
  LEFT OUTER JOIN TableB b ON b.JoinColumn = p.JoinColumn
)
FROM TableA a

Unfortunately trying to use an outer join prompts the error message: Error: Correlated…

Apr 8

The pain of Avro schemas

Recently I’ve been working with Apache Avro and I must say, the development experience leaves a lot to be desired. It’s very easy to make a mistake whilst writing the json and the avrogen tool will happily create the C# classes without giving any kind of warning. …

Avro

2 min read

Mar 2

Using the kafka dotnet client on Apple Silicon

Kafka clients in different programming languages use a common c++ library, which presumably encapsulates the “real” work. As of writing this, the nuget package hasn’t been updated with the arm64 binaries. This causes issues when trying to target dotnet 6.0, as it no longer uses rosetta to emulate x64. …

Kafka

2 min read

Feb 4

Azure DevOps pipeline cheatsheet

Developers can go a long time between having to touch CI/CD pipelines, and it then takes a long time to remember what all the quirks are. Also whilst editing existing pipeline definitions, it can be difficult to reason about why steps, stages or jobs were chosen in the first place…

Azure Devops

2 min read

Jan 26

Database integration tests with docker

I like to have integration tests for these reasons: The repository or SQL logic can be complex. Hand-rolled SQL isn’t type-checked, so we rely on automated testing. In-memory mocks conceal database vendor-specific details that don’t show up until running against the real thing. It’s very easy to create database integration…

Integration Testing

2 min read

Jan 20

Using a proxy to capture http requests in dotnet

Sometimes you want to use a proxy to capture http requests, for example to check if a specific header is being set. This is easy to accomplish using the postman application, which you probably already have installed. Set the environment variable “HTTP_PROXY” to “http://localhost:5555". You can add this in your localSettings.json file for example. Enable capturing requests in postman. You can see from the screenshot that port 5555 is being used, the same port as set in step 1. I’ve chosen to record requests in a collection called “Captured”, created explicitly for this purpose.

Software Development

1 min read

Using a proxy to capture http requests in dotnet

Set the environment variable “HTTP_PROXY” to “http://localhost:5555". You can add this in your localSettings.json file for example.
Enable capturing requests in postman. You can see from the screenshot that port 5555 is being used, the same port as set in step 1. I’ve chosen to record requests in a collection called “Captured”, created explicitly for this purpose.

The modal for enabling capture of requests in postman

When you start your application you should see the requests being added to the collection you selected in step 2.

Jan 8

Implementing the KotZen parser-combinator

This is a follow-up post to the introduction to KotZen, a parser-combinator library. The project is on github and the docs are hosted on netlify if you want to browse the exported classes and functions. With this post I’ll show a breakdown of how it works from the bottom-up. …

Kotlin

6 min read

Jan 7

Introducing KotZen — a parser-combinator for kotlin

I’ve long been enamored with parser-combinator libraries, such as Sprache. In fact I’ve managed to use them in professional and personal projects on several occasions now. The first was at a brokerage firm, where traders would write quotes in a specific shorthand to customers, which was very terse in order…

Kotlin

2 min read

Jan 6

Transactionally update Azure App Configuration

We’ve elected to use azure app configuration in order to be able to adjust settings without needing to redeploy the whole app. What we didn’t want to do, was adjust these values via a web UI. This would be risky, because we would lose change control and versioning history. The…

Azure

2 min read

Dec 22, 2021

Software Rewrites – The Forever War

Recently I asked myself how much of my career has been spent rewriting “legacy” software. At first I thought the answer is probably a good four out of ten years, although if I’m really being honest with myself, it’s what I’ve been doing almost the whole time. Allow me to…

Software Development

4 min read