Oblique

The wrong way to block “Sign in with Google”

Maya Kaczorowski — Thu, 27 Aug 2026 14:16:30 GMT

If you’re a startup, chances are you’re using Google Workspace, and also using Google as your identity provider. This is convenient, since your team can use single sign-on to “sign in with Google” to most SaaS apps you use: they get access without needing to manage a username and password, and the login is tied to their account, so that when they leave, they lose access.

Well, kind of. Google is an identity provider, and it’s a good authenticator, but a bad governor. It'll tell an app who someone is when they log in, and that they’re part of your org. It won't tell you what they can do once they're in, and it won't reliably take it away: you can't guarantee that when someone leaves their access is gone, since you likely can't enforce that Google is used for SSO in every SaaS app. None of which means you should buy something else on day zero. Google is still the right IdP when you’re starting out. (Google is also often special-cased in SaaS app pricing, like it’s not a ‘real’ authenticator worth charging more for 🤷‍♀️.) You should just be aware of what it is and isn’t doing for you.

Google works well for ensuring users can access the apps that they need to. But one of the reasons you want an IdP is to help you manage what folks have access to in your organization, including limiting which apps are used — so why don't you block users from logging into arbitrary apps with their Google ID? (If you're just learning that was possible now… I think we know why.)

You can block your users from signing into third-party apps with their Google Workspace account by changing the setting for “Unconfigured third-party apps”. "Unconfigured" means anything you haven't already marked Trusted, Limited, or Specific Google data in the app access control list. (If you haven’t spent much time inside Google Admin, most likely all of your apps are unconfigured.) There are three configuration options for handling unconfigured apps:

Allow users to access any third-party apps. This is the default, and allows your users to access anything, and grant any app any access.
Allow only apps that ask for basic sign-in info. Users can sign into any apps that just want a name, email address, and profile pic. Any app asking for more is blocked. But this isn’t a filter: if an app wants Drive access, the user doesn't get a degraded version of that app, they just don't get the app at all.
Block users from accessing any third-party apps you haven’t explicitly configured. This blocks all OAuth scopes, including sign-in.

Configure settings for “Unconfigured third-party apps” in the Google Admin console

Why bother to restrict any apps at all? The main reason is that most users can't distinguish (or don't care) about the difference between logging in and granting an app access to everything in their mailbox. No one reads the consent screen.

The second reason is duller: you should, in theory, know what apps your users are using. Not so you can block anyone, but because you need that list for vendor security reviews and on- and offboarding. The OAuth grants in Google Admin are a starting point for that list, not the list. It tells you what was granted, not what's being used, and it lags changes by a couple of days.

If you block users from new apps, they hit this “Access blocked” screen. Technically, it's a request screen: they can hit “Request Access”, and you can add a custom message telling them what to do. But what happens next is a black box to the user. They submit a request, and they have no idea who it goes to, how long it’ll take to approve, or how long it’ll take to propagate. It might be easier and faster to just hit the back button and sign up with a username and password, or a personal email. It might be easier to just give up on getting access at all. Users tend to pick the path of least resistance.

After the OAuth consent screen to log in with their Google account, users hit this screen where they can request access to an app

So what should you do, if anything? As always, it depends on your risk tolerance, but I'd suggest you pick between the first two options. Allowing only apps that ask for basic sign-in info isn't blocking those grants, it’s blocking those apps, which only makes sense if you have someone reviewing those blocked apps. If you’re not, just leave the free-for-all default.

Either way, you'll end up with apps in your environment you'd rather not have — but if you still let users sign in to them, at least you'll have a list of them, you'll know who authorized them, and you'll know they're behind SSO. And that's better than not knowing about those apps at all.

Hacking SAML with Claude Code

Eric Chiang — Wed, 19 Aug 2026 18:10:00 GMT

I’ve worked with the SAML protocol ever since I made the unfortunate decision to add support to Dex almost a decade ago. SAML is often described as actively dangerous due to the brain melting technology that is XML Digital Signatures, and every SAML library I’ve ever interacted with has eventually been found to have a full authentication bypass. My own implementation in Dex resulted in multiple vulnerabilities.

A couple months ago, I used this experience to give a talk on the history of SAML vulnerabilities at the Sec Nerds SF conference. This was an opportunity for me to rant a little, but also brush up on bypass techniques and the latest research in the area.

Around this time, I was reading Niels Provos’s “Finding Zero-Days with Any Model” and started to wonder what would happen if I took my research on SAML vulnerabilities and gave it to a harness. I applied to Anthropic’s Cyber Verification Program to remove the guardrails on my account, and after being accepted, I had access to an Opus model that could write exploits fed by my research and prompting.

I like to claim that SAML is a bad protocol, but could I prove it?

During my spare time over the course of a month or so, I attempted to hack every SAML implementation I could find. While I wasn’t able to structurally break the protocol (much to my chagrin), there were plenty of bugs, many of which are still live today.

Find me vulnerabilities, Claude

One of the core differences between asking Claude to code and asking it to hack is scale. Even in a large codebase, the amount of context needed to track down an individual bug or implement a feature can be relatively small. On the other hand, hacking is about being exhaustive, and going until you find a bug or run out of threads to chase.

To account for this, essentially all “hacking harnesses” (of which there are many) coalesce around a few core primitives:

Break up work between multiple agents
Provide storage for intermediate results that agents can use for direction
Prioritize, scope, and dedupe the work to be done

I built the same. (Check out the source code at: github.com/oblique-security/saml-research)

Building the harness was easy. What I was still surprised by (even despite all the coverage of Mythos, OpenAI/Hugging Face, and model escapes) was that I didn’t really need to teach Claude Opus how to hack. Giving Claude a corpus of vulnerabilities often devolved into it trying to find replicas of those exact issues in other libraries. I found better results by providing a threat model and then letting Claude do its own exploration. This is similar to my experience with agentic coding: as the models get better, you provide fewer specific instructions, and more general guidance.

My pipeline settled on two core phases. A “gadget” phase attempting to find weird behavior in the underlying libraries, and a “findings” phase that combines gadgets then confirms them by writing an end-to-end exploit. These are written to JSONL files, which different steps perform operations on. For example, a step that rejects proposed findings that are out-of-scope before spending tokens to prove them.

Here’s an example of a gadget that Claude found in Node’s xml-crypto’s handling of processing instructions:

{
  "id": "g-0005",
  "title": "C14N DIFFERENTIAL -- processing instructions are FLATTENED to their data text: `` canonicalizes to the bare characters `echo 1;`",
  "impacts": [
    "s-0108",
    "s-0126",
    "s-0210"
  ],
  "status": "confirmed"
}

Which the finding phase later turned into an email truncation bypass:


  
    
    
-      not-an-admin@example.com
+      admin@example.com

Investigating every SAML implementation

It was now time to take my Claude Max 20x plan and see what my pipeline could find.

Since 2020, there’s been a bypass in a major SAML library about once a quarter. GitHub Enterprise alone had four across 2024-2025 (CVE-2024-4985, CVE-2024-6800, CVE-2024-9487, CVE-2025-23369). Essentially all of these stem from components interpreting XML in subtly different ways, and mistakenly processing unsigned data as if it was verified. Today many libraries have adopted hardened APIs that attempt to defend against these kinds of attacks.

While I wasn’t able to set the Internet ablaze, I was able to find full authentication bypasses in four different projects:

Authentik: Injecting a comment in NameID can be used to truncate to another user’s account and authenticate as them (CVE-2026-57580)
PHP litesaml/lightsaml: Signature wrapping on Response message (CVE-2026-63182)
OneUptime: Signature wrapping on Response message (OneUptime/oneuptime#2949)
Java’s saml-client: Signature wrapping on Response message (justinbleach/saml-client#149)

Again, if you’re ever going to hand roll an SAML implementation, don’t (maybe even just use OpenID Connect instead?).

The Authentik vulnerability was the most interesting because eight different independent researchers reported it at the same time. Clearly, I’m not the only one hacking with AI.

After looking at many top level “login as anyone” attacks, I started poking around other parts of the protocol. In SAML, a “SAML Response” containing a user’s email is signed, but technically any message can have a signature, AuthnRequest, AttributeQuery, and LogoutRequest being the most commonly implemented. These other messages rarely get the kind of security attention the SAML Response does, and I found signature bypasses in twelve projects (and counting) for these components, resulting in information disclosure and arbitrary logouts issues. An example that’s been publicly reported in the TypeScript project samlify.

Finally, denial-of-service vectors. SAML fundamentally requires processing arbitrary XML documents POST’d over the internet, and almost every major SAML library was susceptible to out-of-memory conditions from unauthenticated requests. I was able to get patches accepted for the Go ecosystem, but all Python and Node libraries have active DoS issues that are currently unfixed.

Go’s xmldsig had a quadratic memory allocation during signature validation due to a missing check. JavaScript’s xmldom causes similar allocations (though the report is currently still private) impacting all Node libraries. Python packages don’t filter the set of transforms they pass to libxmlsec1, so an XSLT template can cause the libraries to generate an arbitrarily large document:

The state of vulnerability research in 2026 is bleak. Many of my reports landed on with maintainers who either didn’t respond, or if they did, were clearly overwhelmed. If being an open source maintainer wasn’t already a thankless job, they’re now getting flooded with both genuine and slop vulnerability reports, thanks to AI. One company I reported an issue to insisted I submit a screen recording to filter out generated reports (which I refused to do, but due to the impact of the vuln, they still responded, fixed it, and I got a CVE).

One of the most of-the-times interactions I had was with the OneUptime project, who… let’s say… have embraced AI. After no response to a private report for a month, I opened a public issue, only for a very large PR to merge later that day claiming to have fixed it with no human acknowledgement. I then found a new bypass, and a new large fixed appeared. After the third bypass, the large PRs seemed to arrive at something secure:

It turns out LLMs are also bad at writing secure SAML code.

Ultimately, I picked a stopping point not because I had run out of bugs, but because the appetite for these reports wasn’t there.

So what did we learn?

SAML has been around for long enough that we’ve grown past the point where a stiff breeze will shake out numerous “login as anyone” issues, though the exception continues to be newly written implementations. Projects without the benefit of many years of issues will reimplement the same bugs and have the same kinds of bypasses, so I would again stress: do not roll your own SAML implementation.

I hope this kind of meta analysis becomes more popular. Security Engineers will often debate things like whether or not JWT are good or bad. Having the ability to scan all of GitHub and come up with concrete vulnerabilities is motivating in a way that CVE data and anecdotes aren’t. If you think a protocol has issues, you now have the tools to prove it.

If you’re ever thinking of integrating SAML into a product, be sure to run an LLM evaluation against your implementation. And if you want some common vulnerability patterns to look for, feel free to reach out to me!

What's still annoying about SOC 2 in 2026

Maya Kaczorowski — Tue, 28 Jul 2026 18:41:31 GMT

We just hit that milestone that every startup dreams of: we got back our SOC 2 Type II report from our auditor! Try not to be jealous. This covers the suitability of the design and operating effectiveness of our controls relevant to the Security and Confidentiality trust services criteria, throughout the period January 1 2026 to March 31 2026. If you’re a customer or prospect, get a copy from our trust center.

This isn’t my first SOC 2 rodeo. But it’s certainly different as the primary individual responsible for getting our ducks in a row. Although our CTO Eric handles engineering and production security, I cover everything else, from corporate security to policies to HR to operations. And I’m one of the lucky ones! For so many startups, the whole compliance team is one person (at least I have someone to talk to).

In the past several years, the best practices on SOC 2 haven’t really changed (see for example Latacora’s starting seven), but the tools certainly have. Compliance platforms are more expansive (and cheaper) than ever — leading to some, shall we say, questionable products. It’s not just the compliance platforms; the availability and quality of free security tooling, whether for vulnerability scanning, or SIEMs, is the best it’s ever been.

Given these advancements, what can I possibly share from our experience that might help you? What’s still annoying about getting a SOC 2 audit in 2026?

Access reviews are still manual

You need to regularly conduct access reviews (CC6.3) to ensure you didn’t miss any changes to users’ permissions when they moved roles, or you didn’t leave any permissions lying around when an employee left. In practice, what that means is pulling together a list of all of your apps, all the users in those apps (and the roles they have), and reviewing it piece by piece.

This is painful today not because that’s hard, but because it’s manual. You likely don’t have an authoritative list of all of the apps you use, and even if you did, they’re likely not all connected to your SSO provider, and even if they are, they don’t necessarily all enforce SSO and block other users from accessing it. So you do need to actually go into each of the apps to get this data, and dump it somewhere like a spreadsheet to centralize it so you can review it. What we did was create a template in Notion of all of our apps and their user pages (and admin pages, if separate), so we could just run the same process every time. Making this repeatable is what saves you.

The value we got from doing these reviews was variable, and really depended on what the vendor surfaced in terms of information about users in the app. First off, verifying user settings like MFA was incredibly confusing. Some apps have SSO (and MFA enforced via the IdP) but still show that the org doesn’t enforce MFA. Some apps like Google Workspace report that the user doesn’t have MFA enabled because they have been using a passkey. The only way we could disambiguate findings was by reviewing them app by app.

Conversely, having role information was quite helpful, and where we had the most ‘findings’. We found some roles were inconsistently granted in engineering apps that might not be used day-to-day, like Figma, Semgrep, and Sentry. These were generally for employees who had joined since the last review cycle: we hadn’t been consistent about what role they were being granted.

We didn’t find any red flags, and that’s not surprising at our size, but this review will only get more complex as we grow. And that’s part of why we’re working on this now at Oblique: to make access reviews easier for other orgs like us.

Evidence is still scattered across tools

A lot of SOC 2 is about scanning your infrastructure and code for vulnerabilities (CC7.1) and responding to those findings in a prompt manner — but maybe even more importantly, proving that you did. Our logs mostly route into RunReveal, but security alerts go to Slack, where we get notifications from RunReveal and Semgrep (Google Cloud alerts go to Security Command Center instead, since they'd bury everything else). The problem is that Slack was where alerts went to be seen, not fixed. Our actual work lives in Linear, so anything that didn't become a Linear ticket never got picked up or resolved.

Since there’s still an element of triage involved, we respond differently depending on the tool: logging alerts are immediately reviewed by an engineer, whereas vulnerability scanning alerts require regular checks to see if there’s anything worth fixing amidst the noise.

And since we have disjoint tools for scanning different parts of our stack — and they don’t feed findings into a central location — getting evidence of the work we’ve done to remediate these issues means we need to pull that information from several tools. Unlike access reviews, which are a snapshot of a point in time, logs have to provide historical context: they need to show not only when an issue was found but also when it was fixed. If you’re running a scan in your PR checks, it may not keep the historical logs you need. GitHub Actions keeps logs for 90 days by default, and you can raise that to 400 days, but you have to know to change this setting before you lose all your old logs. Even if you extend this, you might be fine for a 3 month audit period, but you certainly don’t want to discover this at the end of your 12 month audit period.

Unsurprisingly, the hard part of vulnerability management is the human process.

Some controls are still vestigial

Some parts of SOC 2 are from a bygone era, but it doesn’t mean you don’t need to follow them (you still do). One of the few things our auditor found we were missing evidence for is a device disposal tracker (CC6.5). We don’t have removable physical media, no one has left, and no laptop has been retired… so this has literally never happened. Even when we eventually do have to retire a laptop, it’s FileVault-encrypted with the org-wide key in Apple Business, so we can easily remotely wipe it. But don’t worry, we still created the tracker. (It’s a Linear issue label.)

Good security is still the best shortcut

Completing a SOC 2 audit, or any audit, is only proof that you can follow instructions for doing the minimum. But as a security company, it’s not enough for us to meet just the minimum requirements — we aim to be excellent at security. Luckily, investing in security also helps us simplify the work we need to do for our audit.

Our biggest audit win was not one we chose for compliance: adopting simple architecture. Boring infrastructure choices and deterministic tests make an environment easier to maintain — and easier to audit. We have a monorepo, use managed services to host our platform (Google Cloud Run and Cloud SQL), and use distroless container images as our base. And, we manage all of our Google Cloud infrastructure only as code in Terraform, and wherever possible where our other vendors allow it.

A lot of our security posture comes from engineering decisions we'd have made regardless of compliance:

We use Go for our backend, which is a memory-safe language. We also use protocol buffers and gRPC for all of our APIs, which gives us strong typing between our frontend and backend. We can even ensure our SQL statements are free from user-controlled input.
Checking in our configuration also allows us to write linters and tests against our production setup. We run PR-time checks to validate development flags can’t make their way to production.
There's no shell into our production infrastructure. Cloud Run doesn't offer one, and we didn't build one because we don’t need it: everything in our production environment is managed via Terraform, and we have a test environment for testing new configs. For the audit, that means there's no privileged access to review: no bastion, no session logs, no break-glass SSH procedure to write down and test. And without write permissions, our engineers have limited access through alternative means like Cloud Console.
Similarly, there’s no direct access to our production database. You can’t run ad hoc queries against prod. There's no direct DB credential or connection to lock down, audit, or worry about someone forgetting to revoke.

Completing an audit gives you the assurance that your controls are correctly implemented and work in practice. It forces you to think about your security holistically, and consider if there's an even better, more secure way to be doing the same task. But that warm fuzzy feeling can't keep us jaded security professionals going for long — we know an audit isn't proof you're secure, because you never really are. Still, we hope that sharing how we think about (and keep improving) our security is useful to you too. We'll keep at it.

If you need a copy of our SOC 2 report or want more information on our security controls, check out our trust center.

What a great users page looks like

Maya Kaczorowski — Fri, 17 Jul 2026 16:11:46 GMT

Every application has a list of users. But outside of adding or removing users when employees join, move, or leave, if you’re looking at that user list, chances are you’re there because of an audit, or an action you take for an audit like regularly reviewing access to your systems. It would be nice to be able to look in your identity provider to see all the access information at once, but unfortunately, the data that exists in your identity provider likely isn’t right — because even with single sign-on, unless you’re able to block any other authentication method, you could still have other users in the app.

We’ve been working on building user access reviews at Oblique, which means I’ve been staring at a lot of application user lists, both in admin interfaces and their exportable CSVs. Here’s what I wish every application had.

Somewhere in the admin settings of your app you should be able to find a list of users in your application and their roles. This is most likely a list of all users, but if your application is widely used within an organization, this might just be a list of admins, with the full list of users available elsewhere.

That list, at minimum, should:

Uniquely identify individuals, which means not only using their names but also their emails
Tell you the role the user has, and whether that role includes admin powers
Tell you some information about the security of the user’s login method, such as that they log in via SSO, they have 2FA enabled, or they have a passkey
Tell you the status of the user (are they an active user?) and ideally when they last logged in or used the application
Make this information easily exportable, via a CSV download

It’s really not that much. If I could have the perfect table, it would be filtered only to active users and show: user, email, role, 2FA, and last login time. (If you only have SSO for your app — good job! — you get a free pass, but it’s still nice if you make that obvious from this page.) That can fit nicely in any UI and still show me everything I need to know at a glance. And it’s important to actually show this information, and not, say, truncate a user’s inconveniently long name (Johan Brandhorst-Satzkorn isn’t your colleague?), or sneakily paginate so you can’t easily see the full list.

Here’s the admin table we have in Oblique, showing: user, email, whether they are the organization owner, and when they last logged in. Oblique requires SSO, so other authentication settings like 2FA and passkeys are enforced by the identity provider.

If you wanted to add more, a CSV export could additionally include any service accounts, deactivated users, and invited users; as well as when the user joined or was deactivated. You could also include information about the login method. And please, name the CSV something sensible, like myapp-admins-date and not users. I have so many files named users(2) on my desktop.

Name,Email,Admin role,Admin since,Organization owner,Last login,Terraform managed
Claude Shannon,claudeshannon@rhombic.dev,Admin,2026-07-14T21:26:20.545Z,No,,No
…
Maya Kaczorowski,maya@rhombic.dev,Admin,2026-07-14T21:26:20.461Z,Yes,2026-07-14T22:59:48.261Z,No
…

And here’s the corresponding admin table export from Oblique, which also includes when the user became an admin, their admin role, and if their admin user is managed via Terraform.

Don't structure this export in such a way that I need a pivot table to get to the info I really want. For example, if your application has teams and users can be part of multiple teams, then that should be its own table, not in this one. As when dealing with any data set, don’t give a column multiple meanings. Don’t have a single field for “status” which conflates deactivated, bot, and role fields all in one. And don’t have a data set where some columns depend on other columns without indicating the relationship, like showing me 2FA information even when the user can only log in via passkey. Sadly, I’ve seen all of these.

I’d also highly suggest you make all of your admin pages, including this users page, have a URL you can easily link users to. Ideally, this would be one that doesn’t depend on an organizational ID, though I understand that for much more complicated products with nested sets of permissions this can be hard to avoid.

As a product manager, I’ve always enjoyed making the little boring everyday parts of security almost enjoyable, where you’re pleasantly surprised to see that someone thought about the exact situation you’re in right now. Sharing best practices like this benefits us all, especially now that it’s so easy to implement. You could just point your agent at this post directly — you really have no excuse.

What is separation of duties, anyways?

Maya Kaczorowski — Tue, 26 May 2026 08:01:15 GMT

As we build more complex approval and review requirements in Oblique, I’ve been digging into compliance frameworks. We want to make sure that we can support any reasonable requirement an organization might have, especially if they’re being mandated to follow it. Compliance frameworks rarely tell you exactly what to do — most requirements are short, vague, and left to interpretation. So it's worth digging into what's actually being asked.

Separation of duties is one that comes up occasionally. It also turns out to be one of the most misunderstood. For example, NIST CSF 2.0 requires that access permissions "incorporate the principles of least privilege and separation of duties." Which sounds reasonable until you try to figure out what it actually means.

Separation (or segregation) of duties is the principle that a risky workflow shouldn’t be able to be completed by a single individual. The prototypical example is that the individual who processes payments shouldn’t be the one approving them. And this makes sense in fraud generally, but that’s really the only example you see come up in the literature.

When you dig into the payments processing example to understand the underlying thinking, you can break it down a few ways:

Don’t let the same individual have both roles
Don’t let the same individual have both roles at the same time
Don’t let the same individual have both roles for the same transaction

The industry tends to focus on the first interpretation (flagging the "toxic" combination of roles — yes, that's really what it's called), when what we actually care about is the third: that the individual cannot unilaterally approve their own transaction. (Some people shouldn't be able to approve transactions at all, but that's not properly implementing least privilege — it has nothing to do with separation of duties.)

How does this apply to a more modern workflow? You can't easily configure your repo so Alice only reviews code and Bob only opens PRs (at least, not without custom roles). Nor can you control that Alice doesn’t have both of these permissions at the same time. But branch protection rules make sure that Alice can’t approve her own PR — which is what actually matters. And we see similar controls in banking, so that individuals can’t approve their own expenses.

This isn't really a ‘traditional’ access management problem. The systems where these high value or risky transactions take place (like for payment processing or production changes) already require a second reviewer. The control isn’t implemented using roles, but rather implemented at the transaction level, and decides whether or not an individual can approve their own transactions.

Implementing separation of duties as blocking toxic role combinations is a reasonable control. But it really exposes that you don't have the control you actually want: to limit self-approval at the transaction level. And if a SaaS app doesn't give you a way to block self-approval, I have no reason to think its role-level controls are any better.

The security policy rollout survival guide

Maya Kaczorowski — Wed, 20 May 2026 16:13:20 GMT

This blog post is a written version of a talk that our cofounder Maya gave at BSidesSeattle in February. You can also watch the recording and get the slides.

Security teams are constantly defining and rolling out new policies. Defining the policy is the easy part. The hard part is knowing what's actually possible to implement, and then implementing it.

Here's the map for how to navigate everything that needs to happen between defining a policy and it going ‘live’ in your environment.

Get input

There's something driving this policy forward, whether it's a compliance requirement, a recent incident, or just a security best practice, but there's usually room for interpretation between the policy (what you want to be true, like peer review on production changes) and the control (how you enforce it, like branch protection rules). The first move is to get input from the people who'll have to live with your controls, because they control your policy’s destiny.

For prod policies, your stakeholders are engineering, product, and SRE. They want to know, “how long will this take to implement?” and more importantly, “who gets paged when it breaks?”. For corp policies, your primary stakeholder is IT. Too often security throws a policy over the wall to IT to figure out how to implement, plus handle frontline response for user issues. What IT asks is, “how do we resolve issues that come up?”, but what they mean is, “are we responsible for this?”. Just like getting buy-in for any other change in a large organization, be proactive, and answer questions before they come up.

Communicate the change

When you’re making a change, communicate way more than you think you need to, and in multiple ways: email, Slack, and all-hands meetings. Before the rollout, during the rollout, and after it's a success. Tell people what's changing, when, and, most importantly, why (compliance counts as a why). People will put up with the friction if they understand the reason, and may be more tempted to circumvent it when they don’t.

Run a pilot

You wouldn't roll out a change to prod without testing it, would you? Don't roll out a change to your users without testing it either. There will be edge cases, controls that fail in unexpected situations — and you really need user feedback.

A good pilot is made up of a few elements:

A representative set of participants: A diverse group of real users, who have time to engage with you. Don’t pick the team next to you because it’s easy, and definitely don’t pick the team shipping a feature next quarter because they’ll be too stressed to give you good feedback. Look for a range of roles, devices, and workflows: macOS, Linux, and Windows; employees, interns, and contractors; engineering, support, and sales.
An appropriate timeline: Enough time to roll out a change and get feedback on it, but importantly, long enough that the control is actually tested. A policy requiring access approvals every 90 days can’t be tested in two weeks.
A support channel: A way for users to report issues, escalate bugs, and give feedback.
Clear success criteria: Success for a pilot is a go/no-go decision. You can usually ‘go’ once you know your rollout will achieve the desired outcome, blocking bugs are addressed, user feedback isn’t overwhelmingly negative, and the support burden is manageable.

Iterate

It’s possible (in fact, highly likely) that you will get feedback during the pilot that changes your implementation. That’s the point.

Use the pilot to find bugs, identify edge cases, and — most importantly — test that your controls actually work. If users find a way around your control, your policy doesn't work.

When you triage feedback, ultimately, the question is: is this blocking rollout? Is this affecting all or only some users or platforms? Is this a bug or an improvement? Can we handle this case differently, and still get the same outcome?

Roll out policy

I have a few pet peeves on doing policy enforcement ‘well’.

First, make enforcement of a policy visible. Measuring whether people comply changes if they do. If you want users to take action, nothing beats showing them how they compare to their peers.

Second, the person who sets the policy should be the one enforcing it. They need to feel the pain (that’s why you did a pilot!). Any ongoing user friction should hurt the policy owner more than the user. (The one exception is compliance requirements… you can’t make your auditor feel the pain for you. They are, however, very good at making you feel pain.)

Lastly, exceptions. There will always be exceptions. Always! What matters is knowing this ahead of time, so that you can build these in and have a strategy for managing exceptions over time from the start. In addition to the obvious ones (specific departments, employee types, platforms), you'll also likely have exceptions for users with disabilities, users in certain countries, demo devices, and automated workflows. There is no one size fits all. What’s important is deciding if these are out of scope, or just have an exception. Either way, managing them is never done.

After all that work, you've rolled out a policy. Did the policy do what you said it would? Probably, because you got input, piloted it, iterated on feedback, and then managed the inevitable exceptions. The harder question is whether you kept your users' goodwill while doing it. Because there's always a next one.

Generating an MCP server in Go

Eric Chiang — Thu, 14 May 2026 17:45:00 GMT

MCP is table stakes for products these days. Everything’s AI, and agents will query your data one way or another.

As we’ve built Oblique, a core principle is not having internal APIs. Anything a user is able to do in the UI should be possible through Terraform or REST - and now, MCP. We generate tons of bindings from our gRPC interface with the explicit intent that as new features are added, they make their way to all of our integration points.

Wait, isn’t MCP dead?

There’s an infinite number of ways to expose capabilities to agents. Command line tools, SQL interfaces, transpiling to TypeScript. If there exists some means of querying or modifying data, someone has experimented with it as an MCP alternative.

Regardless of the tech, the model needs to understand the capabilities exposed to it and how to exercise them. For a command line tool, this means keeping your “--help” output accurate. For MCP, this is tool descriptions and input schemas. Fundamentally if you’re providing primitives to an agent, you’re in the business of keeping the documentation for those interfaces up to date and debugging when things go wrong.

Where MCP is irreplaceable are the increasing options for autonomous agents. Your Claude Code Routines and Codex Automations of the world. Sure, you may choose to implement raw tool calls through a custom harness for your internal workflows, or shell out to a binary on your laptop. But if you want to expose data to a customer workflow, you have to speak MCP. It’s become the lowest common denominator for working with agents.

Tool descriptions and schemas

Descriptions and schemas are critical for tool discovery and informing the model how to construct tool calls. A majority of the issues we hit boiled down to incorrect plumbing of comments, or typos in our API docs. For example, we had a bug where enum fields were commented but their values weren’t.

As we covered in our previous post, we generate as much of our server tooling as possible from Protobuf. In this case, we leveraged Go’s protoreflect package to produce tool definitions for the Go MCP SDK. This means we consume a proto file:

service Oblique {
  // Fetch a user by name, or using the alias "users/me" to return the currently
  // authenticated user.
  rpc GetUser(GetUserRequest) returns (User) {
    option (google.api.http) = {get: "/api/v1/{name=users/*}"};
  }
}
message GetUserRequest {
  // Name of the user of the format "users/{id}". The alias "users/me" is also
  // supported and resolves to the currently authenticated user.
  string name = 1 [
    (google.api.field_behavior) = REQUIRED
  ];
}
// A user represents a human user in the directory.
message User {
  // Format: `users/{id}`
  string name = 1 [(google.api.field_behavior) = IDENTIFIER];
  // Primary display name of the user.
  string display_name = 2 [(google.api.field_behavior) = REQUIRED];
  // Primary email of the user.
  string email = 3 [(google.api.field_behavior) = REQUIRED];
  // ...
}

…and spit out a tool description:

{
	"annotations": {
		"readOnlyHint": true,
		"title": "GetUser"
	},
	"name": "GetUser",
	"description": "Fetch a user by name, or using the alias \"users/me\" to return the currently authenticated user.",
	"inputSchema": {
		"type": "object",
		"properties": {
			"name": {
				"type": "string",
				"description": "Name of the user of the format \"users/{id}\". The alias \"users/me\" is also supported and resolves to the currently authenticated user."
			}
		},
		"required": [
			"name"
		]
	},
	"outputSchema": {
		"type": "object",
		"description": "A user represents a human user in the directory",
		"properties": {
			"name": {
				"type": "string",
				"description": "Format: `users/{id}`"
			},
			"displayName": {
				"type": "string",
				"description": "Primary display name of the user."
			},
			"email": {
				"type": "string",
				"description": "Primary email of the user."
			}
		},
		"required": [
			"name",
			"displayName",
			"email"
		]
	}
}

We’ve consistently found that human, handwritten comments are tremendously powerful at directing models and outperform anything generated. Both in the context of an AGENTS.md file or tool descriptions. This makes some intuitive sense, since if an agent can generate a comment, it can likely derive that information anyway.

The challenge here is accessing source code comments which aren’t available in files produced by the Go proto plugin. To work around this, our generation scripts output a descriptor file and embed it in our Go server. That file is then parsed with the protodesc package to access comments as we walk the message descriptors:

//go:embed descriptor_set.binpb
var descriptor []byte

var descriptorFiles *protoregistry.Files

func init() {
	opts := protodesc.FileOptions{AllowUnresolvable: true}
	set := &descriptorpb.FileDescriptorSet{}
	if err := proto.Unmarshal(descriptor, set); err != nil {
		panic("parsing descriptor: " + err.Error())
	}
	files, err := opts.NewFiles(set)
	if err != nil {
		panic("resolving files: " + err.Error())
	}
	descriptorFiles = files
}

type direction int

const (
	input  direction = 0
	output direction = 1
)

func newTool(md protoreflect.MethodDescriptor) (*mcp.Tool, error) {
	desc, err := descriptorFiles.FindDescriptorByName(fullName)
	if err != nil {
		return "", fmt.Errorf("finding descriptor by name: %s: %v", fullName, err)
	}
	description := desc.ParentFile().SourceLocations().ByDescriptor(desc).LeadingComments

	isDestructive := strings.HasPrefix(string(md.Name()), "Delete")
	readonlyPrefixes := []string{"Get", "List", "BatchGet"}
	isReadOnly := false
	for _, p := range readonlyPrefixes {
		if isReadOnly = strings.HasPrefix(string(md.Name()), p); isReadOnly {
			break
		}
	}

	inputSchema, err := schemaForMessage(md.Input(), input)
	if err != nil {
		return nil, fmt.Errorf("generating schema for input: %v", err)
	}
	outputSchema, err := schemaForMessage(md.Output(), output)
	if err != nil {
		return nil, fmt.Errorf("generating schema for output: %v", err)
	}

	toolName := renameTools()
	return &mcp.Tool{
		Name:         string(md.Name()),
		Description:  description,
		InputSchema:  inputSchema,
		OutputSchema: outputSchema,
		Annotations: &mcp.ToolAnnotations{
			Title:           toolName,
			DestructiveHint: &isDestructive,
			ReadOnlyHint:    isReadOnly,
		},
	}, nil
}

Our schema generation also filters fields based on the context (the code was a bit big for this post). If a message is used for tool input, fields with an OUTPUT_ONLY annotation are ignored by the generator. We also remove metadata fields from our API objects that aren’t critical for the model. Consider the following result from our REST API (185 tokens):

{
    "name": "users/ekqzuevxt544jl9i",
    "createTime": "2026-05-12T20:51:28.504018Z",
    "updateTime": "2026-05-12T20:51:28.512136Z",
    "deleteTime": null,
    "displayName": "Eric Chiang",
    "email": "eric@rhombic.dev",
    "secondaryEmails": [
        "eric@obliquesecurity.com",
        "eric@oblique.security"
    ],
    "title": "Software Engineer",
    "manager": "users/sx1qezek79nqyrb0",
    "pictureUri": "https://avatars.githubusercontent.com/u/2342749",
    "directReportCount": 2,
    "totalReportCount": 2
}

On output we use protoreflect to clear common fields (“createTime”, “updateTime”), as well as message-specific fields that aren’t as relevant to an MCP client. In this case our user object consumes half as many tokens (97) as its API equivalent:

{
    "name": "users/ekqzuevxt544jl9i",
    "displayName": "Eric Chiang",
    "email": "eric@rhombic.dev",
    "secondaryEmails": [
        "eric@obliquesecurity.com",
        "eric@oblique.security"
    ],
    "title": "Software Engineer",
    "manager": "users/sx1qezek79nqyrb0"
}

Debugging performance

The most common bug report from our initial test was “the model is confused by X.”

If you’re at a big company, you have an entire team dedicated to evaluations and prompt engineering. For this particular feature, it was just me debugging where the agent was getting tripped up, and trying to ensure new changes didn’t degrade previous performance.

One of the early features we implemented was a developer mode running our MCP server over stdio. This paired with simulated data allowed us to run prompts in a sandbox. Our first iteration was to call Claude Code’s programmatic mode with an MCP configuration and stream the results:

SERVER_BIN="$PWD/bin/oblique-server"
go build -o "$SERVER_BIN" ./cmd/oblique-server
SYSTEM_PROMPT="$PWD/mcp/eval-systemprompt.txt"
TMPDIR="$( mktemp -d )"
MCP_CONFIG="$TMPDIR/mcp_config.json"
cat >"$MCP_CONFIG" <

Claude Code’s streaming JSON format logs a ton of internals about the harness to help understand what’s going on. We can see the results of tool search, to help determine if a tool’s description needs improvement:

{
  "type": "user",
  "tool_use_result": {
    "matches": [
      "mcp__oblique__GetUser",
      "mcp__oblique__SearchUsers",
      "mcp__oblique__SearchTeamMembers",
      "mcp__oblique__SearchTeams"
    ],
    "query": "select:mcp__oblique__GetUser,mcp__oblique__SearchUsers,mcp__oblique__SearchTeamMembers,mcp__oblique__SearchTeams"
  }
}

Or inspect successful and unsuccessful tool calls, which can point to deficiencies in our schema comments:

{
  "type": "assistant",
  "message": {
    "type": "message",
    "role": "assistant",
    "content": [
      {
        "type": "tool_use",
        "name": "mcp__oblique__SearchUsers",
        "input": {
          "filter": "email=\"eric@oblique.security\""
        }
      }
    ]
  }
}

We took that initial bash script and wrote a ~300-line Go program that consumes a library of prompts, calls Claude Code in parallel for each one, and parses the output using Go’s streaming JSON support. For every prompt, the program produces a report including the set of tools that were loaded and the tool calls:

- prompt: What teams am I on?
  tool_matches:
    - mcp__oblique__GetUserProfile
    - mcp__oblique__SearchTeams
    - mcp__oblique__SearchTeamMembers
    - mcp__oblique__BatchGetTeams
  tool_calls:
    - name: mcp__oblique__SearchTeamMembers
      input: '{"parent":"teams/-","filter":"member=\"users/me\""}'
    - name: mcp__oblique__BatchGetTeams
      input: '{"names":["teams/engineering","teams/spanish-conversation-group","teams/puzzle-box"]}'
  tool_errors: []
  result: |-
    You're on 3 teams:

    - **Engineering**
    - **Spanish conversation group** — ¡Hola! ¿Cómo estás? ¡Ven a charlar con nosotros los viernes por la mañana! All levels welcome
    - **Puzzle Box** — Make better puzzles

This view was invaluable to see if changes were steering the model in the right direction. Any time we get a bug report, we now add a prompt to our evals that attempts to replicate it, and are very quickly able to see where the model is getting stuck.

`So what did we learn?`

Generating our MCP server this way has produced a great feedback loop for our API documentation. Good descriptions improve agent understanding as much as humans. We made dozens of tweaks to our API comments as clients hit issues, which in turn are pulled into generated bindings for developers (and coding agents), and into future hosted API docs.

While not everything in our REST API cleanly fits into MCP, keeping them coupled with conditional generation logic lets us ensure our MCP server gets treated the same as any other surface in the product.



The performance bug hiding in our billing settings
Johan Brandhorst-Satzkorn — Tue, 05 May 2026 16:00:00 GMT
At Oblique, we run our backend on Google Cloud Run. It's a great fit for what we need: a Go binary, a Postgres database, and a handful of background jobs that sync data from upstream identity providers and integrations. Cloud Run handles the lifecycle, we ship containers, everyone is happy.
Until we started noticing slow database query performance.
This is the story of how it took us two months on and off building out our amazing observability stack: adding traces, switching tracing backends, enabling query insights on our database, chasing locks, hypothesizing about VPC networking… and ultimately learning that Cloud Run was throttling our CPU outside of HTTP requests. We didn’t need all of that, and a single one-line annotation on our Cloud Run service would have fixed the entire problem from day one:
metadata:
  annotations:
    run.googleapis.com/cpu-throttling: "false"
The rest of this post is about how we got there, why we didn't get there sooner, and why we’re still happy we did all the work.
The symptoms
Our backend has two kinds of work. The first is the obvious kind: HTTP requests from the frontend, served by a Connect handler running each request in a goroutine. The second kind is background work: goroutines are periodically kicked off by the same binary that serves the frontend, and these goroutines handle anything performance intensive: syncing data from identity providers and other integrations, recomputing group memberships, running garbage collection, and so on.
The HTTP request endpoints seemed to be unaffected, but the background jobs were unreasonably slow, even considering how much work they were doing.
We were seeing single SQL queries that should have hit an index and returned in tens of milliseconds take five seconds. Background sync jobs that locally took a couple of seconds were stretching to ten minutes or more. Our database connection pool (4 connections per instance) was constantly saturated, with pool acquisition calls themselves taking multiple seconds.
Our first instinct was to suspect the database queries, so that’s where we started.
Step 1: dive into the data
The first thing we did was look at the traces we already had. We'd been exporting traces to Google Cloud Trace from our Go backend for a while. Looking at our existing data, something was clearly wrong. Some background jobs were taking 5-10 minutes in production, but our existing traces weren't telling us why. We could see that a span was slow, but we couldn't see what the goroutine was actually doing during that time, what error it hit, or what query it was running. Our spans had almost no metadata. We could see the shape of the slowness without seeing the cause.
So, we started enriching our traces. GCP recently added support for consuming the OpenTelemetry trace format (OTLP) directly. We rolled out OpenTelemetry behind a feature flag, both to allow us to run trace collection locally, and to lean into the OTel ecosystem of libraries for observability data. It allowed us to run a Jaeger collector locally and inspect traces in real time while running the application against a local Postgres instance. If we could reproduce the slowness locally with better tooling, we'd have a much faster debug loop.
The results were immediate and clarifying: not a single trace locally took longer than one second against a local postgres DB. Whatever was happening, it wasn't a property of our code or our queries in isolation. Since our production environment is different from our local development environment, and identical code running in both environments exhibited different characteristics, that’s where the problem had to be.
Step 2: chase every hypothesis
Once we turned on OTel in production, we could see actual error messages on failed spans. We could see the pgx driver's pool.Acquire as its own span. We could see individual queries with their durations. We started forming hypotheses. We had a lot of them.
Hypothesis 1: too many SQL calls. One of our background jobs was making roughly 1,500 SQL calls per run, mostly because we weren't batching anything. We started batching the SELECTs and INSERTs. This was the kind of cleanup we'd been meaning to do for a while, and it did help, but not enough to explain the multi-second outliers. So, that wasn’t the issue.
Hypothesis 2: lock contention in the database. We thought maybe write locks from one job were blocking reads in another. We wrote some Postgres queries to look for blocked queries, looked at the Cloud SQL metrics, and turned on Query Insights. That helped us understand what part of the query was taking a long time, and the trend of queries over time. Something that was very confusing was just how big the deviation between individual identical queries was: taking anywhere from 200ms to 40s to run.
Another curious artifact that Query Insights allowed us to see was that we seemed to spend a lot of time downloading the result rows from the database:
We couldn’t find any evidence of lock contention, so it wasn’t that either.
Hypothesis 3: connection pool saturation. With only 4 connections per instance, any slow query starves the pool. This was clearly part of the problem. When you can see pool.Acquire taking seconds, the pool is your bottleneck. But it was a symptom, not a cause. Something was making individual queries slow, and the pool was just amplifying it. We were on the right track, but the pool itself wasn’t the issue.
Hypothesis 4: network latency between Cloud Run and Cloud SQL. The working theory at this point was that network latency was adding just enough overhead to cause individual connections to pile up. This became a leading hypothesis after we turned on Query Insights and saw that the majority of the query time was spent in downloading the result rows.
We then tried switching our Cloud SQL connection to using a private network connection, but it made no noticeable difference. A change we made at the same time however did make a surprising difference.
Cloud Run execution environments
Cloud Run has two execution environments. The first-generation environment is gVisor-based, a userspace kernel that intercepts syscalls. The second-generation environment runs your container in a Linux microVM with a real kernel. If you don't explicitly specify an environment, Cloud Run picks one for you based on the features your service uses.
The execution environments documentation recommends using the second generation environment if your service has CPU-intensive workloads or could benefit from faster network performance, so we pinned our containers to that environment, because we were willing to try anything at this point. After we’d made the switch, query performance improved:
Pinning to the second generation environment was clearly an improvement. However, shortly after celebrating, we discovered to our horror that the query metrics went back up again. Not as bad as before, but still not where we wanted it to be.
We still didn’t understand what was going on, but without a clear way forward we went back to working on regular application features for a while, with the nagging feeling that something still wasn’t right.
Two weeks later
Background jobs were still slower than they had any business being. We had made some progress, and still believed that the networking between our containers and the database was somehow cursed, since the change to the second generation environment had improved things slightly. I had just submitted a PR to increase the minimum CPU allocated to our instances to 2 CPUs when we started discussing the possibility of CPU throttling. Reading through the Cloud Run billing settings docs, one line jumped out:
The CPU allocated to all containers in an idle instance depends on the configured billing settings.
Wait. Idle instances? Is an instance with no in-flight HTTP request considered "idle," even if it's running goroutines doing background work?
Cloud Run has two billing modes: request-based and instance-based. Under request-based billing (the default), your container's CPU is throttled when no HTTP request is being processed. Not "scaled down." Throttled. The degree of throttling depends on the runtime environment. The reason we’d seen a speedup with the switch to the second generation environment wasn’t because of network performance, it was because it can’t throttle as aggressively as the first generation.
Our entire background processing system runs as scheduled goroutines that aren't tied to HTTP requests at all. Every single one of those goroutines had been running under heavy CPU throttling the entire time. The fix is an annotation, which translates to setting your instances to instance-based billing:
metadata:
  annotations:
    run.googleapis.com/cpu-throttling: "false"
Now your container gets full CPU all the time, regardless of whether it's currently handling an HTTP request. You pay slightly more, because you're billed for the wall-clock time your instance exists rather than just the time it's serving requests, but for a service like ours, which runs background jobs continuously anyway, that’s the behavior we expect.
We rolled out the change and watched the metrics dashboards. The 99th percentile query latency on our background jobs collapsed:
Background jobs that had been taking tens of minutes now finished in under thirty seconds, the remaining time limited mostly by integration APIs rate limiting. Pool acquisition contention dropped to nothing because individual queries were now finishing as fast as they should have all along. All of the batching work, all of the trace improvements, all of the VPC investigation, none of it was the actual fix. The fix was one annotation that, in retrospect, we should have set on day one.
Why we don’t regret the work we put in
The temptation, after a debugging journey like this, is to conclude that we wasted two months on and off on work we didn’t need. We didn't.
OpenTelemetry was worth it. The richer span metadata, the ability to run a local trace collector, the integration ecosystem. All of it survived past the fix and is still paying dividends. When the next mystery slow path shows up, we'll see it faster.
Query Insights is worth keeping on. Enabling it is free, you only pay for data retention. It didn't help us find this particular bug, but it may help us narrow down issues with slow running queries quicker in the future. It’s better to have it before you need it. If you're running on Cloud SQL and you don't have it on, turn it on.
The batching work was worth doing. 1,500 sequential SELECTs in a single background job was a problem regardless of CPU throttling. We would have had to fix that eventually. We just fixed it earlier than we needed to.
Pinning the execution environment was worth doing. The Cloud Run scheduler picking a different environment under your feet makes for an unpredictable metric baseline. Even if the environment generation difference hadn't mattered for latency, the fact that the choice is invisible and non-deterministic is enough reason to pin it explicitly.
The annotation we should have set on day one. If you're running on Cloud Run and you have any kind of in-application background work (goroutines you start from a request handler, scheduled jobs, async queue consumers, anything), set run.googleapis.com/cpu-throttling: "false". The performance characteristics of CPU-throttled background work are so bad, and so non-obviously bad, that the marginal cost of always-on CPU is basically irrelevant compared to the cost of debugging this for two months.
Why didn’t we find it sooner?
We had good observability, better than most. We were actively looking. Why did we spend two months on and off on hypotheses that weren't the answer before stumbling onto the one that was?
There are three reasons.
The first is that the symptoms looked exactly like a database problem. Multi-second SQL queries, connection pool exhaustion, lock contention candidates. When you see those symptoms, your professional instinct is to investigate the database. And the database was, in some sense, not actually slow. It was the client that was being suspended mid-flight by a cgroup, holding a connection while it slept. From the database's perspective, the client just took a really long time to send or receive the next byte. From our traces' perspective, the query took five seconds. Same data, completely different interpretation.
The second is that "Cloud Run throttles your CPU when no request is in flight" is a non-obvious property of the platform. It is documented. It just isn't documented in the places you'd look while you're debugging slow code. The billing settings page describes the two billing modes accurately, but it's framed entirely as a cost question, not a performance one. The general development tips page mentions in a single sentence that you should use instance-based billing for background activity, but it's one bullet among dozens of unrelated tips. The 2021 launch blog post for the always-on CPU feature is the clearest explanation of the model, and it's a four-year-old blog post. None of these are the pages you read when your database query performance is bad.
We aren't the only people to have hit this. After we figured it out, I traded notes with another Cloud Run user who'd run into the same problem from a different angle. They were doing background processing triggered by Pub/Sub, and they'd convinced themselves it couldn't be a CPU allocation issue, because the docs say CPU is allocated "during request processing time" and they had a request in flight the whole time the processing was happening. What they'd missed was that they were acknowledging the Pub/Sub message early to avoid redelivery, and continuing the actual work in the background afterwards. The HTTP request had returned. Cloud Run considered the instance idle. The work that the user thought of as "request processing" was, from the platform's perspective, background work, and it was being throttled accordingly. The phrase "request processing time" is doing a lot of work in those docs, and it doesn't mean what most people reading it would assume.
The third reason is subtle and unexpected: Go handled this absurdly under-resourced situation with grace. Our service was being asked to do real work on roughly 5% of a vCPU, and it kept working. Goroutines parked, the scheduler kept multiplexing, the runtime kept making forward progress. Queries finished eventually. Background jobs completed eventually. Nothing crashed, nothing OOM'd, nothing fell over. The system was, in a meaningful sense, operating correctly the whole time. It was just slow.
That's a real strength of the Go runtime, and we're glad we wrote the backend in Go. But it also presented us with a steady-state of "everything still works, it's just bad," which is one of the hardest failure modes to debug, because it doesn't trigger any of the signals operators normally rely on. The dashboards were green. The error rate was fine. The service was up. It was just inexplicably slow, in ways that wasn’t visible to users, for months.
If there's a moral to this story, it's that observability is necessary but not sufficient. We could see exactly what was happening (pool waits, slow queries, sluggish goroutines) and the picture was so consistent with "the database is overloaded" that we kept looking at the database. Better data doesn't fix that on its own. You also need to keep poking at your assumptions about the platform you're running on.
Clearly the real moral of the story though, is to read the billing docs.


How Oblique handles code review etiquette
Jenny Zhang — Thu, 30 Apr 2026 15:00:00 GMT
Code reviews can be a source of immense anxiety for engineering teams. Many of us have had bad experiences with nitpicking, bike-shedding, or that’s-not-how-I-would-have-done-it-itis. This is such a common pain point that there are entire research projects built around code review anxiety and how to reduce it.
The age of agentic coding brings a whole new layer to this dynamic: a fellow engineer disappearing into a cave and coming back with 12,000 lines of code you were expected to review that same day used to be a cautionary horror story; now that could be an average Tuesday with a poorly supervised agent. Do you let it by with a cursory LGTM? Do you throw another agent at it to do the review for you? Do you even still have coworkers?
It doesn’t have to be like this.
The fun thing about building a startup is you also get to build an engineering team from scratch, with your own engineering culture: one that hopefully avoids the worst of the friction from past jobs. We’re small enough now that we can talk as a group about how things should be done, but we won’t stay this size forever. Given how quickly norms are changing across the industry, we wanted to take some time to write down how we navigate code reviews at Oblique so that we’re all on the same page about how we want to collaborate as a team.
Here’s where our internal guidance landed.
Know what the goal is
The explicit goal of code review is to ensure high-quality engineering. The implicit goal, which is just as important, is to ensure high-quality communication among team members. Even on a small team it is easy to lose track of how our product and codebase are changing, and code reviews are a crucial tool for sharing context and thinking.
That means both authors and reviewers should treat code reviews primarily as an act of collaborative learning. Do not send slop PRs, which are disrespectful of the reviewer’s time. Do not send slop reviews, which are disrespectful of the author’s effort.
PRs are not a good venue for debating architecture, and should not replace engineering discussions and design documents. Good communication starts long before the code gets written. If you as a reviewer are surprised by the contents of PR, that’s a sign that there’s a gap in the team’s communication somewhere. If a reviewer has concerns about the fundamental approach of a change, that should be expressed as a top-level comment or, better yet, through a conversation with the author.
Optimize for the reviewer
We all have different styles of working, but at the end of the day, someone else has to be able to read and understand your code. Oftentimes that someone else is you, six months (or days) from now.
Consider how diffs read and attempt to minimize them. Before sending a PR for review, authors should read through their own change in the GitHub UI—you’d be amazed at what mistakes you’ll suddenly spot when you’re looking at your own code in a different context. Reduce changes that aren’t relevant to the core logic, ensure tests have clear names, comment complicated blocks, follow the existing style of the surrounding code.
PR size is more of an art than a science. Keep PRs self-contained, brief, and reviewable. Large PRs can become functionally impossible to review, while extremely small ones lose the context of the change. Always attempt to compose the smallest PR that stands on its own. Wherever possible, work iteratively and merge code behind feature flags instead of developing out-of-tree.
Author responsibilities
We trust one another to use the best tools for the job, and we also expect that authors are responsible for all code they send. This means that authors should read, review, and understand any changes generated by coding agents. Authors are not required to disclose their use of AI tools, and reviewers should have the same standards regardless of how a change was created.
What is important to disclose is the maturity and intent of your change. If you threw a prototype together just to explore a concept and you don’t care if it ever gets merged, regardless of whether that prototype was handcrafted by you or by Claude, that should be expressed clearly in the PR description so that reviewers know what kind of feedback to provide.
Agentic tools have a tendency to produce functions with long names and flowery multi-line comments. These often have little substance despite their verbosity. Reviewers will always appreciate a hand written test name, or a few comments on the key lines over a spaghetti of helpers.
Every commit to main must work. This means if a backend change introduces something that breaks the frontend or vice-versa, it must be feature-flagged or the exception must be handled. Authors alone bear the risks for large branches and conflicts that occur because of them. Moving fast is not an excuse for breaking things.
Reviewer responsibilities
Prioritize code reviews over writing code. A reviewer that lets their queue get out of hand blocks the whole team. Each reviewer is expected to set up their notification workflow such that they can provide timely responses. (At Oblique, we rely heavily on the Slack and Linear integrations for GitHub.)
Trust your teammates to take your feedback seriously. Where possible, approve the PR with comments to avoid a second round-trip.
Ask for help when you need it. If a PR feels too unwieldy to understand, it’s okay (encouraged even!) to ask for it to be broken up into smaller pieces. Everyone benefits from a better shared understanding of the logic. Alternatively, don’t underestimate the power of asking to walk through a review together.
It’s also okay to ask an agent to help you digest the PR, as long as you remember that the end goal is for you to understand the PR. If you’re just copying and pasting an agent’s review output into GitHub, you’re not exactly adding anything the author couldn’t have done themselves. 
Reviews should focus on APIs and the tests for those APIs. Don’t overindex on style or internal details, unless those have a noticeable impact on performance, security, or readability. Critiques should be constructive and substantive - there are many different ways of accomplishing the same thing, and someone else choosing a different path than you does not necessarily mean they’re wrong. If there is a substantive reason to align on a specific style, add a linter or update the relevant AGENTS.md files.
Always remember that your teammate is on the receiving end of the review. A good reviewer takes a moment or two to digest the change as a whole before providing feedback. Avoid knee-jerk reactions and brigading, and consider the tone of your comments. When in doubt, discuss face-to-face.
Conclusion
Our team is still small, so these expectations seem obvious to all of us now. The good thing about writing this down is that when we fall short of our ideals, we have documented guidance that we can revisit to course-correct. It also means we have a historical snapshot of our thinking, which makes it easier to see if our old practices are no longer serving us. Software engineering has always been a dynamic practice, and these days it’s changing faster than ever. Now we have a baseline.


The $0 security stack
Maya Kaczorowski — Mon, 06 Apr 2026 17:28:18 GMT
In building a tool that determines what access is allowed in your environment and which integrates with your most sensitive internal systems (your identity provider and your HRIS), we have to care about security. Not just because we like it, and not just for compliance, though compliance is often the motivation to get started with security.
For many SaaS tools, the security features you need (like SSO, SCIM, or audit logs) are often on the highest price tier, because they’re "enterprise" features. You need to pay for security features — but in recent years, there’s been a different approach with security vendors. Between open source and product-led growth, security doesn’t have to cost a lot. In fact, for a team like ours, it costs nothing.
We just completed our first SOC2 audit period, and as part of that, we set up our initial security stack. We’ve set up literally world-class security tools, without it costing us a cent.
Here’s our $0 security stack.
Semgrep for SAST and SCA
We use Semgrep for code analysis and supply chain analysis (CC8.1). It’s free for up to 10 contributors, so it’s free for us. It’s made some valuable suggestions, such as forcing a higher minimum version of TLS or using a different library. The AI-based triage means that issues are automatically triaged as false positives based on context in the codebase, or dependencies that aren’t reachable aren’t prioritized.
TruffleHog for secret scanning
We use TruffleHog on every commit to make sure we don’t accidentally leak secrets. We initially ran the open source Action on every commit, but now we run it daily to verify we haven’t regressed. It hasn’t caught anything, which is a good thing!
RunReveal for SIEM
We use RunReveal as our SIEM, which ingests logs from our infrastructure (GCP, Cloudflare, and GitHub). Once we set up log streaming, we turned on the built-in detections (CC7.2), and hooked it up as an alert to a private #siem Slack channel. RunReveal’s Community tier supports 5 data sources and generous retention. It found some misconfigurations we had in the first 24h, but since then the only time it’s gone off was over the holidays when it thought our log streaming was broken since no one was deploying 😂.
Sublime Security for email security
We connected Sublime Security to our Google Workspace, and have it send alerts to the #siem Slack channel as well. Their Core tier is free for up to 100 mailboxes.
We assume that phishing attempts (that we absolutely get sent) will occasionally succeed, and so rely on SSO and/or MFA across all systems to limit the damage of a single click, but also use email security tools to flag and mitigate suspicious emails as an additional defense. I strongly prefer this to doing phishing simulations (which might get brought up in CC2.2), and which we know don’t work.
Sublime detects more than Google out of the box, but we still get a lot more false positives than ideal, so we are now testing out another solution in parallel.
Apple Business for MDM (now free)
Unfortunately, I lied, there is one security tool we pay paid for: an MDM. (Everyone else, please don’t charge me!). It may seem crazy to have an MDM at a company our size, but… try deploying one later and you’ll regret it. (And I refuse to deploy a compliance vendor’s osquery shim with no enforcement. If that’s the solution then we could just deploy osquery ourselves… which also does not spark joy.)
We use Apple Business (formerly Apple Business Essentials) to enforce disk encryption (CC6.7), password lock (CC6.1), force updates (CC7.1) — and share WiFi creds. (Note that Mac devices have XProtect anti-malware (CC6.8), and it can’t be disabled.) It was already less than the cost of Notion… and now it’s free.
As a founder, I get it — every dollar matters. But if you're already paying for compliance, the incremental cost of doing security correctly is smaller than you think. "We can't afford security" is almost never true in 2026, even if you’re a startup. There's more excellent, free tooling than you realize. You just have to look.
(Also, it’s not lost on me that we don’t have a free tier of Oblique yet for access requests and access reviews (CC6.2, CC6.3). Give us a bit more time.)


Passkey PRFs for end-to-end encryption
Eric Chiang — Wed, 25 Feb 2026 17:23:00 GMT
End-to-end encrypted apps have always had to reckon with users losing their encryption keys. If you brick your phone, you just assume you're never getting your Signal chats back. But as E2EE has found its way into consumer services, there are more seamless backup and syncs of encrypted data between devices and even into clients like web browsers.
The problem is humans are still bad at picking strong passwords, or memorizing 256 bits of random data.
This coming weekend, I’ll be speaking at BSidesSeattle on how WhatsApp, Signal, and X (Twitter) leverage hardware security modules to encrypt data with a relatively weak passphrase or PIN (Saturday at 3:30pm in Track 4!). What didn't make the cut for that talk is the next generation of schemes of deriving encryption keys from passkeys.
Passkeys are a newish technology based on the security key protocols in browsers and mobile apps. As opposed to a physical hardware token, they are magically synced by your OS credential store or password manager (iCloud Keychain, Google Password Manager, Windows Hello, 1Password, etc.), and replace your password rather than act as a second factor.
As passkey syncing has dramatically improved over the last few years, services like 1Password, Bitwarden, and Confer are leveraging an extension for deriving cryptographic material from passkeys. Today, if you enable WhatsApp encrypted chat backups, by default your backups are protected with a passkey, instead of a password that you have to memorize:
This post covers passkey pseudo-random functions (PRFs), and their use in end-to-end encryption.
Passkey PRFs
Passkeys are extremely limited in the primitives they expose. Through the browser, you can only request a signature over a structured challenge. There's no API for signing arbitrary data or decryption, much less key derivation.
Using an underlying API originally intended for disk decryption, WebAuthn supports a “prf” extension where the client combines a seed (called a “salt”) with a per-credential private value. This produces a random-looking but deterministic result that can be used to derive other cryptographic material like encryption keys.
To enable this extension, pass a salt to the “extensions.prf.eval.first” argument for a registration or authentication request:
// Register a passkey with the server and generate a PRF output.
const cred = await navigator.credentials.create({
  publicKey: {
    challenge, // Provided by the server.
    rp: {
      id: rpId,
      name: rpName,
    },
    user: {
      id: userId, // Provided by the server.
      displayName: username,
      name: username,
    },
    pubKeyCredParams: [
      { type: "public-key", alg: -7 },
      { type: "public-key", alg: -257 },
    ],
    extensions: {
      prf: {
        eval: {
          first: salt, // Provide a salt for a deterministic output
        },
      },
    },
  },
});
The salt can be chosen based on the needs of an application. A static value is fine since the PRF is unique per-credential, so two users (or the same user on different sites) will always produce different outputs. Per-user salts can be useful for supporting rotation of key material for more advanced needs.
One benefit of a static salt is that it can be provided during the login challenge, before you know which user is authenticating:
// Use a static salt value.
const salt = new TextEncoder().encode("my-sites-static-salt").buffer;

// Login the user and generate a PRF output all in one go.
const cred = await navigator.credentials.get({
  publicKey: {
    challenge, // Provided by the server.
    rpId,
    extensions: {
      prf: {
        eval: {
          first: salt,
        },
      },
    },
  },
});
A second salt can also be passed to the extension as a credential rotation primitive:
// From the Mozilla docs
//
// https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API/WebAuthn_extensions#prf
({
  extensions: {
    prf: {
      eval: {
        first: currentSessionKey, // salt for current session
        second: nextSessionKey, // salt for next session
      },
    },
  },
});
The PRF output
The credential returned by the registration or authentication phase will contain a PRF result field.
const pubKey = cred as PublicKeyCredential;
const resp = pubKey.response as AuthenticatorAssertionResponse;
const ext = pubKey.getClientExtensionResults();
// Pseudo-random value
const { first } = ext.prf.results;
The client can then use the output to derive key material as input to an envelope encryption scheme or Double Ratchet algorithm. In a more straightforward case, the client can feed the result into the Web Crypto API to encrypt data directly:
const prfKey = await crypto.subtle.importKey(
  "raw",
  ext.prf.results.first,
  "HKDF",
  false, // Not extractable
  ["deriveKey"],
);

// Derive an encryption key with a standard key derivation function.

// Salt for the HKDF. NOT the passkey PRF.
const hkdfSalt = crypto.getRandomValues(new Uint8Array(12));
const secretKey = await crypto.subtle.deriveKey(
  {
    name: "HKDF",
    hash: "SHA-256",
    salt: hkdfSalt.buffer,
    info: new TextEncoder().encode("note-encryption-key"),
  },
  prfKey,
  { name: "AES-GCM", length: 256 },
  false,
  ["encrypt", "decrypt"],
);

// An "initialization vector" is a public value that MUST be unique per
// encryption event with a given symmetric AES key. Best practice is to
// generate it randomly for every call to "encrypt".
const iv = crypto.getRandomValues(new Uint8Array(16));
// Encrypt with AES-GCM.
const ciphertext = await crypto.subtle.encrypt(
  { name: "AES-GCM", iv },
  secretKey,
  new TextEncoder().encode(data), // Data to encrypt.
);

// Send encrypted data to the server to store as well as public metadata
// (IV and HKDF salt).
const encryptedData = new Uint8Array([
  ...iv,
  ...hkdfSalt,
  ...new Uint8Array(ciphertext)]).buffer;
Later, the client can decrypt the encrypted data held by the server using the same PRF output:
// Receive the "encryptedData" back from the server.

// Extract the IV, HKDF salt, and ciphertext.
const iv = encryptedData.slice(0, 16);
const hkdfSalt = encryptedData.slice(16, 16 + 12);
const ciphertext = encryptedData.slice(16 + 12);

const secretKey = await crypto.subtle.deriveKey(
  {
    name: "HKDF",
    hash: "SHA-256",
    salt: hkdfSalt.buffer,
    info: new TextEncoder().encode("note-encryption-key"),
  },
  prfKey, // Output from PRF.
  { name: "AES-GCM", length: 256 },
  false,
  ["encrypt", "decrypt"],
);

const plaintext = await crypto.subtle.decrypt(
  { name: "AES-GCM", iv },
  secretKey,
  ciphertext,
);
Putting it together
As part of this post, we open-sourced a full demo with a React frontend and Go backend to run locally. You can find the code on GitHub:
https://github.com/oblique-security/webauthn-prf-demo
Unknown block type "videoEntry", specify a component for it in the `components.types` option
Passkey PRFs are an incredibly interesting primitive for cryptographic operations on a mobile app, or in a browser. This isn’t just limited to symmetric encryption. You could imagine seeding asymmetric key generation for SSH connections, or signing application prompts for audit logs. If you’re willing to put up with a little bit of JavaScript cryptography, there’s now a robust means to drive these schemes with key material that magically syncs between devices.


Fit access controls to your org, not the other way around
Maya Kaczorowski — Thu, 19 Feb 2026 21:20:13 GMT
Turns out, you mostly ship your org chart. That's Conway's Law, and it applies to access controls just as much as it applies to software architecture, for better or worse.
Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations. —  Melvin E. Conway
Our job at Oblique isn't to tell you how your organization should be set up. It's to give you the tools to work with how it already is.
We’ve talked with a lot of organizations about access management. Although they all want to express access controls slightly differently, they’re all more similar than they are different. Your company may be special, but your org chart really isn’t all that special. For example, many organizations have a concept of a small ad hoc team working together on the same project that doesn’t necessarily map to where they sit in the org chart, whether they call it a “squad”, “pod”, or something else. The same concepts keep cropping up across organizations, although the specific implementations might differ.
Organizations mostly use one of three ways to group users together to decide who should be granted access to something:
By department or function. For example, everyone in sales needs access to Salesforce. This is closest in spirit to role-based access control, but ironically it is often implemented using attributes, with an attribute of department or title. (In Oblique, these are attribute-based groups.)
By manager or reporting chain. For example, everyone who reports to Samantha needs access to Zendesk, because Samantha runs customer success, which could include multiple departments like support, sales engineering, and DevRel. This is distinct from an attribute — it’s strictly about org structure. If Samantha gets promoted, the access controls follow whoever takes over her old job. (In Oblique, these are reporting groups.)
By project. For example, everyone working on the MCP server needs access to PostHog, even though that project has engineering, as well as product, design, docs, marketing, and SRE. Often, this is tied to a specific resource rather than an application, like access to a particular Snowflake table rather than the Snowflake app. It can also be tied to a communication channel or group. (In Oblique, these are team groups.)
What an individual needs access to depends on a mix of all three group types. Although the specifics depend on the culture and org structure, almost every organization starts the same way: inherent access tied to employment status, so that all full-time employees get access to basic collaboration tools like Google Workspace and Slack. Additional access depends on what system and data is the source of truth for departments and teams — sometimes it’s the reporting chain, sometimes it’s a department or cost center attribute, and sometimes it’s a completely ad hoc, manually-managed team.
This complexity — of using multiple data sources and group types to manage access — is why access controls end up being so complicated. There are many stakeholders involved in org structure design, but IT is usually not one of them. But then IT is tasked with using an access control system that doesn’t fit reality, and so the result is a mess: groups based on data that isn’t maintained, and access that’s hard to trace and understand.
The goal with defining groups for access management is not only to make the initial setup sensible (so that you can reason about access), but also to make the system possible to maintain. Your decision depends on what data you have and where its source of truth lives. You need data that’s high fidelity, that someone is already maintaining, and that’s in a system that your access controls can integrate with, like your HRIS.
Your access control model needs to map to how your organization works, not the other way around. There is no single ‘right’ way to set up permissions using these groups. There is only the way that works for your org.


Access requests are a bandaid, not a fix
Maya Kaczorowski — Wed, 11 Feb 2026 18:53:19 GMT
For many IT and security teams, identity feels like a constant treadmill of access changes: add the new hire, remove that contractor, update access to the sales tool, and worst of all, make sure nothing breaks during next Monday’s reorg. To handle the increasing volume of these access changes, the identity industry has gotten very good at handling them quickly, by letting you request elevated privileges in a CLI, or approve access changes in Slack. But you’re still on a treadmill — we’re just making it go faster and faster. What if you want off the treadmill?
The only real security improvements are solving classes of problems
In many areas of security, we’ve learned that fixing classes of problems is the only way we can address them for good. Instead of finding (and fixing) an increasing number of insecure code findings, what if we could reduce the total volume of issues by moving to memory safe languages? Solving a broader class of problems is necessary to reduce the volume of issues.
In vulnerability management, we already had automated patching, with tools like Dependabot or automated OS updates. But there is still an order of magnitude more impact by moving to a minimal base image. It’s not about applying patches faster, it’s about never getting them to begin with. Automated patching helped, but it wasn’t the real fix. Removing unnecessary dependencies was.
We haven’t had that mindset shift — and so that kind of impact — in access management. Access management is still the same: constantly updating and changing access as people’s roles and responsibilities change. It’s still human-intensive work.
Access management is full of bandaids
We’ve kept implementing bandaids to address the fact that our access management in practice doesn’t match it in reality. It makes sense to regularly review access to verify that it’s still correct — because why would you have checked since you first provisioned it? This sanity check was so desperately necessary that quarterly access reviews have become entrenched as compliance requirements across multiple industries, even if many security teams treat them as a checkbox exercise.
The next bandaid was expiring access. If we have access automatically expire, then there’s less to clean up. That makes sense. But expiring access isn’t widely available — most identity providers and applications don’t natively support it — which means it’s not frequently used outside of the highest risk situations, like prod access. Removing access automatically also means you need to think about how to reprovision it when users need it again.
Which brings us to the bandaid of Slack-based access. Rather than making a user file a ticket in Jira or ServiceNow, what if they ask for access where they already are, in Slack? This is a huge improvement in terms of user experience — but it leads to neverending access request tickets, for both the IT team to respond to, and for the user to file. The number of tickets has grown untenable. But faster approvals don’t fix the underlying issue. They just make living with it more tolerable.
Access requests are a bandaid, not a fix
With IT Ops, IT teams are moving from living in a ticket queue to building and automating operational systems. And as more and more IT teams report into the CISO, they’re adopting the security engineering mindset that’s emerged in the past few years. IT teams are using more automation and code than ever before — and they’re looking at their daily piles of access request tickets and asking why there’s not a better solution.
You can only make approvals go so fast if your request flow itself is a bandaid, because you can’t express the policies you actually want. You can’t automate a human — you’ll always be limited in how much you can automate if you explicitly decided to include a human in the loop.
The issue is both technology and process. Rarely do you actually want a line people manager determining who gets access to prod, just like you don’t want them deciding what new project management tool you procure. But when IT and security teams require manager approval as part of an access request, it’s often as a proxy control, to ensure that an intern doesn’t access production, or a contractor doesn’t have visibility to customer data. If you can’t define the controls you actually want, you need to find alternative controls to meet the same requirements. We need better building blocks.
We need expiring access, so that a one-off production debug session doesn't turn into permanent prod access that no one remembers to revoke. We need auto-approvals for known, expected access changes — like access to the customer database related to debugging a specific issue, or as part of an incident — so that routine requests don't clog up the queue, while still keeping a record. (Access requests are still necessary, just not at the scale they're being used today.) And we really, really need deny policies: hard guardrails that prevent access that should never happen, without relying on someone to catch it in review. It's a lot easier to maintain a short list of what’s approved than to audit an ever-growing list of places you might have made a mistake.
Getting off the identity treadmill means getting to fewer tickets over time, not faster tickets. The only way to get there is to have your access controls and policies express what your organization actually wants, and the reality of where it is now, updated automatically over time as things change. That’s what we’re working on here at Oblique.


Go’s synctest is amazing
Eric Chiang — Thu, 05 Feb 2026 16:59:28 GMT
We threw Go’s new “testing/synctest” package at a particularly gnarly part of our codebase and were pleasantly surprised by how effective it was. This post covers the synctest package, its nuances, and how it does much more than speed up your tests.
The main headline for Go 1.25’s synctest is its ability to magically advance time. Tests run in a “bubble” with a fake clock and calls to time.Sleep are virtualized, causing them to seem to run instantly:
// This test runs instantly!
func TestSleep(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		var t1, t2 time.Time
		go func() {
			time.Sleep(time.Second)
			t1 = time.Now()
		}()

		time.Sleep(time.Second * 2)
		t2 = time.Now()
		if t1.After(t2) {
			t.Errorf("Expected t2 (%s) to be after t1 (%s)", t2, t1)
		}
	})
}
Run on the Go Playground
But even more than speed, synctest’s real advantage is being able to deterministically reason about the ordering of events in a test. Let me explain.
Background loops
Our product has a large number of background routines. These can be as simple as deleting expired database rows, or more complex logic like leader election between instances.
These routines take the form of Run functions that call another function in a loop (plus some backoff on failures). This is an abbreviated sample of what that might look like:
type DB struct { /* */ }
func NewDB(ctx context.Context, path string) (*DB, error) { /* */ }
func (db *DB) Close() error { /* */ }
func (db *DB) CreateSession(ctx context.Context, id string, exp time.Time) error { /* */}
func (db *DB) GetSessionExpiry(ctx context.Context, id string) (time.Time, error) { /* */ }
func (db *DB) DeleteExpiredSessions(ctx context.Context, now time.Time) error { /* */ }

// Ummm... how do we test RunDeleteExpiredSessions?

func RunDeleteExpiredSessions(ctx context.Context, db *DB) {
	for {
		select {
		case <-ctx.Done():
			return
		case <-time.After(time.Minute):
			// Run every minute.
			if err := db.DeleteExpiredSessions(ctx, time.Now()); err != nil {
				log.Printf("Deleting expired sessions: %v", err)
			}
		}
	}
}
How do we write a test if we need to wait a minute for the logic to execute? This gets even more complicated with nested loops. Our leader election performs different sub-loops depending on if the current process is the leader or not. What do we do?
synctest is about blocking
synctest advances time when it considers all goroutines within the test “blocked”. There are a few rules, but this largely means channel operations, wait groups, and time methods.
https://pkg.go.dev/testing/synctest#hdr-Blocking
The following test demonstrates this by spawning three goroutines and blocking them on different conditions. The runtime sees that all goroutines are blocked, determines the smallest value passed to time.Sleep(), and advances time exactly enough to unblock that call:
func TestBlocking(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ch := make(chan struct{})
		events := []string{}
		wg := &sync.WaitGroup{}
		wg.Add(1)
		
		doneCh := make(chan struct{})

		go func() {
			<-ch // Blocked on channel read
			events = append(events, "channel")
			close(doneCh)
		}()
		go func() {
			wg.Wait() // Blocked on wait group
			events = append(events, "waitgroup")
			close(ch)
		}()
		go func() {
			time.Sleep(time.Second) // Blocked on time.Sleep
			events = append(events, "sleep")
			wg.Done()
		}()

		<-doneCh      // Also blocked on a channel read
		t.Log(events) // Will print ["sleep", "waitgroup", "channel"]
	})
}
Run on the Go Playground
If there are multiple sleeps, each is unblocked in order until it blocks again or exits. This not only makes the test fast, but allows the use of time for synchronization in a way that would be ill-advised otherwise.
For example, the following loop always increments the counter exactly three times. In a “normal” program, this might occasionally only run twice based on when a goroutine happened to be scheduled.
func TestLoop(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		done := make(chan struct{})
		n := 0
		wg := &sync.WaitGroup{}
		defer wg.Wait()
		wg.Go(func() {
			for {
				select {
				case <-time.After(time.Minute): // Wait for a minute
					n++
				case <-done:
					return
				}
			}
		})

		// Wait for three minutes and a millisecond
		time.Sleep((time.Minute * 3) + time.Millisecond)

		t.Log(n) // The loop always runs exactly three times and this prints "3"
		close(done)
	})
}
Run on the Go Playground
Testing our database code
Putting this together, we can cause RunDeleteExpiredSessions to run its loop by sleeping longer than its call to time.After. Since synctest then waits for the loop to block again, we can also be sure that the call to DeleteExpiredSessions has completed when the test logic picks up again.
func TestRunDeleteExpiredSessions(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		// Create a database.
		ctx, cancel := context.WithCancel(t.Context())
		db, err := NewDB(ctx, filepath.Join(t.TempDir(), "test.db"))
		if err != nil {
			t.Fatalf("Creating test database: %v", err)
		}
		defer db.Close()
		
		// Start the deletion process in an external goroutine.
		wg := sync.WaitGroup{}
		defer wg.Wait()
		wg.Go(func() {
			RunDeleteExpiredSessions(ctx, db)
		})
		defer cancel() // Cause RunDeleteExpiredSessions to exit

		// Create a session that's valid for 30 seconds.
		exp := time.Now().Add(time.Second*30)
		if err := db.CreateSession(ctx, "test-session", exp); err != nil {
			t.Fatalf("Creating session: %v", err)
		}

		// RunDeleteExpiredSessions runs every 1 minute. Block until the loop
		// runs once.
		time.Sleep(time.Minute+time.Second)
		
		// Verify that the garbage collector deleted the session.
		if _, err := db.GetSessionExpiry(ctx, "test-session"); !errors.Is(err, sql.ErrNoRows) {
			t.Errorf("GetSessionExpiry returned unexpected error on expired session: got=%s, want=%s",
				err, sql.ErrNoRows)
		}
	})
}
And this runs in a fraction of a second:
% go test -v 
=== RUN   RunDeleteExpiredSessions
--- PASS: RunDeleteExpiredSessions (0.01s)
PASS
ok  	example	0.340s
We used synctest with our system’s core leader election logic that spawns a dozen of these loops, and it ran perfectly without any changes to the code. We were able to test leader election transitions where a new instance take over from the previous one, and pick arbitrary points in the process to stop and inspect the state of the database to verify.
There are definitely nuances to synctest, such as knowing what does or doesn't block, or ensuring that all the goroutines are cleaned up properly. You can read more about this on Go’s Blog, or the package docs.


Type safe frontend APIs
Johan Brandhorst-Satzkorn — Thu, 29 Jan 2026 20:24:48 GMT
The benefits of schema-driven development have recently become more widely known in the backend. Cross-language RPCs, performance enhancements, compatibility guarantees, and a single source of truth for the API definitions are only some of the benefits. While Protobuf and gRPC provide the means for service-to-service communications, it remains difficult to accomplish this for frontend to backend communications. gRPC is itself not possible to use in the browser, because of its reliance on the obscure Trailer HTTP feature. OpenAPI has been trying to provide an IDL for RESTful services, but support is fragmented (OpenAPIv2 vs OpenAPIv3), the quality of language implementations varies widely and the spec is incredibly broad.
A number of projects have popped up that aim to bridge the gap to the frontend using Protobuf as the IDL, such as:
gRPC-Web
gRPC-Gateway (which I have maintained since 2018)
Twirp
Connect
Many more
Out of all of them, the most interesting one is Connect. Connect was created by Buf, but was donated to the CNFC and now lives alongside gRPC as a community supported open-source project, making it more compelling to companies and independent projects alike. Connect is built on the ES2017 standard and so is compatible with all modern frameworks, with examples in major frameworks such as React, Next.js, Vue, Svelte, Angular, Astro and many more.
Connect also integrates nicely with Tanstack Query, a popular framework-agnostic state management solution. At Oblique, we use the Connect API generators with our React frontend and a Connect-compatible backend using Vanguard, another Buf project. Vanguard isn't necessary, and isn't part of this post. Vanguard also allows us to expose a REST-like JSON API, but that will be a topic of another blog post.
The backend
What does it look like to use? As with any Protobuf API, it starts with a .proto file. For this example, we’ll assume a Go backend, which is what we use at Oblique. Let’s define a basic service:
edition = "2023";

package mycorp.mybackend.v1;

// Important for Go generation, not necessary in general
option go_package = "github.com/mycorp/myproject/gen/mycorp/mybackend/v1;mybackendv1";

service BackendService {
	rpc GetCurrentUser(GetCurrentUserRequest) returns (GetCurrentUserResponse);
}

message GetCurrentUserRequest {}

message GetCurrentUserResponse {
	string user_id = 1;
	string full_name = 2;
	string email = 3;
	string avatar_url = 4;
}
An RPC to get information about the currently logged in user is a very common part of any frontend API. This document shows what it might look like when expressed as a Protobuf RPC.
Lets save this file as mycorp/mybackend/v1/backend.proto. Note how the Protobuf package matches the folder structure we picked. We will use the buf Protobuf compiler to generate our client and server libraries. Install buf using the method of your choice. We start by creating a buf.yaml file to define lint rules and other configuration options:
version: v2
lint:
	use:
    - STANDARD
breaking:
  use:
    - FILE
We can now try running buf build to see that everything is set up correctly:
$ buf build
If nothing happens, everything is working! Running buf build gathers all of our Protobuf files and “compiles” them, ensuring there are no missing imports or syntax errors. To generate client and server libraries, we create a new file: buf.gen.yaml, and define and install the Go plugins we want to use:
version: v2
plugins:
  - local: protoc-gen-go
    out: gen
    opt:
      - paths=source_relative
  - local: protoc-gen-connect-go
    out: gen
    opt:
      - paths=source_relative
We use the paths=source_relative option to generate the Go files in the same folder structure as our Protobuf files use. We can now try running buf generate to see that everything is working:
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
$ go install connectrpc.com/connect/cmd/protoc-gen-connect-go@latest
$ buf generate
We should see two new folders created:
gen/mycorp/mybackend/v1/backend.pb.go
gen/mycorp/mybackend/v1/mybackendv1connect/backend.connect.go
They’re the generated Go files containing the type and service definitions for the services and messages in our Protobuf package. At this point, we will need to make sure we have a go.mod file in place. Using the module name we foreshadowed in our go_package option above, run:
$ go mod init github.com/mycorp/myproject
$ go mod tidy
Now we are ready to implement the backend logic. Create a main.go:
package main

import (
	"context"
	"log"
	"net/http"

	"connectrpc.com/connect"
	mybackendv1 "github.com/mycorp/myproject/gen/mycorp/mybackend/v1"
	"github.com/mycorp/myproject/gen/mycorp/mybackend/v1/mybackendv1connect"
	"google.golang.org/protobuf/proto"
)

type backendService struct {
}

func (s *backendService) GetCurrentUser(ctx context.Context, _ *connect.Request[mybackendv1.GetCurrentUserRequest]) (*connect.Response[mybackendv1.GetCurrentUserResponse], error) {
	return connect.NewResponse(&mybackendv1.GetCurrentUserResponse{
		UserId:    proto.String("12345"),
		FullName:  proto.String("John Doe"),
		Email:     proto.String("john.doe@example.com"),
		AvatarUrl: proto.String("https://example.com/avatar.jpg"),
	}), nil
}

func main() {
	mux := http.NewServeMux()
	mux.Handle(mybackendv1connect.NewBackendServiceHandler(&backendService{}))
	log.Println("Serving on http://localhost:8080")
	if err := http.ListenAndServe("localhost:8080", mux); err != nil && err != http.ErrServerClosed {
		log.Fatalf("failed to start server: %v", err)
	}
}
This is of course just a mockup of the real thing. In a real scenario we’d probably have the backend inspect a user’s cookie and look up the authenticated user in a database of some kind, or reject the request with an error if the cookie isn’t authenticated. The server can be run using:
$ go run main.go
Serving on http://localhost:8080
If we wanted to verify that the server is running, we can just use cURL, since Connect supports regular HTTP/1.1 and the JSON content-type:
$ curl \
	-d '{}' \
	-H 'Content-Type: application/json' \
	-X POST \
	http://localhost:8080/mycorp.mybackend.v1.BackendService/GetCurrentUser 
{"userId":"12345", "fullName":"John Doe", "email":"john.doe@example.com", "avatarUrl":"https://example.com/avatar.jpg"}
Note how the method is always POST, and the path is a combination of Protobuf package, service name and RPC name. Now we are ready to implement the frontend!
The frontend
As mentioned before, Connect has examples for several different frontend frameworks, but we’ll use React for this example, which is also what we use at Oblique. Let’s start with a basic frontend skeleton:
app/
	node_modules/
		...
	dist/
		index.html
	index.tsx
	package.json
	package-lock.json
Use a package manager of your choice to initialize package.json and package-lock.json. Here is dist/index.html:

  
    
    
    
    My app
    
  
  
    
  
And here is index.tsx:
import React from "react";
import ReactDOM from "react-dom/client";

const App: React.FC = () => {
  return Hello world!;
};

ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render(
  
    
  ,
);
To turn our TypeScript file into an index.js that the browser can interpret and execute, we’ll use the esbuild bundler. You can use any bundler of your choice for this step, esbuild is not mandatory.
$ go install github.com/evanw/esbuild/cmd/esbuild@latest
$ cd app && esbuild --bundle --outdir=dist index.tsx
This will produce our dist/index.js. With this dist folder setup, we can serve the app using any HTTP file server, but we’ll use our Go server to serve the app because it’s easy, and avoids the need for CORS. One way to do that is to drop an app.go file into the app folder and then embed the dist folder:
package app

import "embed"

//go:embed dist
var Dist embed.FS
This can be imported as a Go package with the Dist variable being an in-memory embedding of the contents of the dist folder. Lets modify main.go to import this file and serve it when not serving API requests:
package main

import (
	"context"
	"log"
	"net/http"

	"connectrpc.com/connect"
	"github.com/mycorp/myproject/app"
	mybackendv1 "github.com/mycorp/myproject/gen/mycorp/mybackend/v1"
	"github.com/mycorp/myproject/gen/mycorp/mybackend/v1/mybackendv1connect"
	"google.golang.org/protobuf/proto"
)

// backend service definitions, omitted for brevity

func main() {
	mux := http.NewServeMux()
	mux.Handle(mybackendv1connect.NewBackendServiceHandler(&backendService{}))
	distFS, err := fs.Sub(app.Dist, "dist")
	if err != nil {
		log.Fatalf("failed to sub dist from embedded filesystem: %v", err)
	}
	mux.Handle("/", http.FileServerFS(distFS))
	log.Println("Serving on http://localhost:8080")
	if err := http.ListenAndServe("localhost:8080", mux); err != nil && err != http.ErrServerClosed {
		log.Fatalf("failed to start server: %v", err)
	}
}
Now if we serve the backend and visit http://localhost:8080 we are greeted with our rendered React app.
Not the prettiest web app ever made, but it gets the job done. Now that we’ve hosted the frontend, lets try generating the client to let it talk to the backend over Connect. The Protobuf plugin we are looking for is called protoc-gen-es, and there are a few different ways to use it. The easiest way for open-source or non-sensitive code is to use the Buf Schema Registry’s remote plugin. Note that this will send your protobuf files to Buf’s servers to allow them to generate your code for you, so don’t do this for sensitive or private code unless you know that this is OK. For this example, we will use a local plugin installed into our node_modules folder, but we should understand that managing local Protobuf plugins can introduce hard-to-debug issues for local users, such as when the wrong version of a plugin is being used.
Install the npm package @bufbuild/protoc-gen-es using your favorite package manager. This will create a file that we can execute using a shell, node_modules/.bin/protoc-gen-es. We can use this file as a protobuf plugin through our buf.yaml file:
version: v2
plugins:
  - local: protoc-gen-go
    out: gen
    opt:
      - paths=source_relative
  - local: protoc-gen-connect-go
    out: gen
    opt:
      - paths=source_relative
  - local: ./app/node_modules/.bin/protoc-gen-es
    out: app/gen/
    opt:
      - target=ts
Use the option target=ts to generate a TypeScript file. Regenerate the files:
$ buf generate
This will create a new file, app/gen/mycorp/mybackend/v1/backend_pb.ts. Again, note how the file path is based on the Protobuf file structure. The generated file exposes some types and values but is mostly useless on its own. To make use of it, we need to add some more npm packages:
@bufbuild/protobuf
@connectrcp/connect
@connectrcp/connect-web
Now we can update our minimal frontend app to make use of the backend over Connect:
import { createClient } from "@connectrpc/connect";
import { createConnectTransport } from "@connectrpc/connect-web";
import React, { useState } from "react";
import ReactDOM from "react-dom/client";
import {
  BackendService,
  GetCurrentUserResponse,
} from "./gen/mycorp/mybackend/v1/backend_pb";

const client = createClient(
  BackendService,
  createConnectTransport({
    baseUrl: ".",
  }),
);

const App: React.FC = () => {
  const [user, setUser] = useState(null);

  const onClick = async () => {
    const response = await client.getCurrentUser({});
    setUser(response);
  };

  return (
    
      User Dashboard
      
      {user && (
        
          User Details
          
            ID: {user.userId}
          
          
            Name: {user.fullName}
          
          
            Email: {user.email}
          
          
            Avatar URL: {user.avatarUrl}
          
        
      )}
    
  );
};

ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render(
  
    
  ,
);
There’s a lot going on here, so lets break it down. First, we create a client:
const client = createClient(
  BackendService,
  createConnectTransport({
    baseUrl: ".",
  }),
);
This client uses the generated BackendService value and some TypeScript magic to forward the type information. The baseUrl is . since we’re using the Go backend both for hosting the app and the API.
const [user, setUser] = useState(null);

const onClick = async () => {
  const response = await client.getCurrentUser({});
  setUser(response);
};
This is where we make the call to the backend. The {} parameter to getCurrentUser is because there are currently no fields on the input request to getCurrentUser. Naturally, the function is async, so we use await to get the response, so that we can set it on useState.
return (
  
    User Dashboard
    
  
);
We hook up the onClick handler to a button to trigger the call to the backend.
{user && (
  
    User Details
    
      ID: {user.userId}
    
    
      Name: {user.fullName}
    
    
      Email: {user.email}
    
    
      Avatar URL: {user.avatarUrl}
    
  
)}
We conditionally render the user information when the user is available (once the backend function call has returned). Re-bundle:
$ cd app && esbuild --bundle --outdir=dist index.tsx
The final website look:
Unknown block type "videoEntry", specify a component for it in the `components.types` option
Making changes
Whew, that was a lot! As we mentioned at the start, there is quite a bit of boilerplate to get everything set up, but now that all that work is done, making changes, or even adding new APIs becomes so much easier that it’ll all have been worth it. Lets try adding a new input parameter to the GetCurrentUser RPC:
// Rest of Protobuf file omitted for brevity
message GetCurrentUserRequest {
	uint32 avatar_option = 1;
}
We then regenerate the files:
$ buf generate
If we look at our frontend, we can see that we now have the option to specify an avatarOption field in getCurrentUser:
const onClick = async () => {
  const response = await client.getCurrentUser({
    avatarOption: 1,
  });
  setUser(response);
};
The TypeScript compiler helpfully tells us that the type of this field is number, so we can’t accidentally send a string or something else nonsensical. Don’t forget to re-bundle the frontend:
$ cd app && esbuild --bundle --outdir=dist index.tsx
On the backend side, we implement support for a few different avatar options:
func (s *backendService) GetCurrentUser(ctx context.Context, req *connect.Request[mybackendv1.GetCurrentUserRequest]) (*connect.Response[mybackendv1.GetCurrentUserResponse], error) {
	resp := &mybackendv1.GetCurrentUserResponse{
		UserId:    proto.String("12345"),
		FullName:  proto.String("John Doe"),
		Email:     proto.String("john.doe@example.com"),
		AvatarUrl: proto.String("https://example.com/avatar.jpg"),
	}
	switch req.Msg.GetAvatarOption() {
	case 0:
		// Default avatar
	case 1:
		resp.AvatarUrl = proto.String("https://example.com/avatar1.jpg")
	case 2:
		resp.AvatarUrl = proto.String("https://example.com/avatar2.jpg")
	default:
		return nil, connect.NewError(connect.CodeInvalidArgument, nil)
	}
	return connect.NewResponse(resp), nil
}
Now we can run the website again and see that the backend will return the avatar corresponding to option 1!
Adding new RPCs and even new services is similarly simple, and with a single source-of-truth for your API definitions, there is never a question of whether the API is exposed by the backend yet, or what the request format looks like. Connect has changed the game by thinking through and following through on a frontend developer experience that finally gets frontend developers onboard and excited about typed API development.
Further reading
If you want to get started with type safe frontend APIs in your own company or project, the Connect documentation has a lot of information to get you started in the language of your choice.
For generic Protobuf API design, the Google AIPs are a great source of advice and rules that you should read. There is a linter you can use to ensure you are conformant.
Likewise, the Buf style guide and buf lint command help you get started and stay conformat with Protobuf and their best practices recommendations.


The real SSO tax
Maya Kaczorowski — Mon, 08 Dec 2025 18:58:40 GMT
We’re preparing for our upcoming SOC 2 audit here at Oblique. And it’s maddening to realize that it’s 2025, and you still can’t reasonably ensure that your employees are using strong authentication.
What your security team really wants is to understand what applications your employees have access to, and enforce that they use strong authentication to access those applications. That’s why SSO exists in the first place! Since we use Google as our IdP and Apple Business Manager as our MDM, our ideal setup would be: use (and enforce) Google SSO everywhere we can — which can prompt for multi factor authentication — and use a passkey bound to the company-managed device where we can’t.
That sounds reasonable, but it's not what happens in reality.
Where apps do support SSO, you often can't enforce that it's being used. And where they don't support SSO, you often can't ensure strong auth. This gap is the real frustration. The SSO tax shouldn't be about having SSO — it should be about enforcing it. Why did I just spend 15 minutes configuring SAML SSO for our HRIS, if a user can still log in with a username and password? Why did I set up an authenticator app for this login if the app is just going to email me a magic link anyway? We recently had an automated compliance check fail because a user only used a passkey — which doesn’t count as MFA. What is this security theater where we constantly weaken security?
I don't want to pay to have SSO. But I’m willing to pay to enforce the use of SSO.
I understand where this comes from. Password resets and account recovery are huge pains for consumer apps, wasting support teams' time. For both consumer and enterprise apps, you usually want the user to be able to log back in (otherwise, how are you going to hit your growth targets?). In an organization, account recovery also wastes your IT team's time. But the usability of SSO means this shouldn't be a problem, unless a user locks themselves out of their main account. In which case, yes, your IT team really should talk to them before resetting anything.
If you're building an enterprise application, please support SSO — and only SSO — so that it's easy for your customers' IT teams to enforce it (because it's the only option) and rely on MFA controls in their identity provider. You can skip building:
MFA that only supports an authenticator app, not a hardware token or TouchID
MFA that only supports SMS or email-based codes
Strong MFA support that allows downgrading to weaker options
Strong MFA support that isn't required
Username/password login that requires MFA, but doesn't support passkeys
Not only is this complicated to build and maintain, it's actively unhelpful for your customers' security posture.
There's one notable exception: enterprise apps that require employees to use a personal email for continuity, so they can still access the account after leaving the company — like payroll providers or equity management tools. In those cases, please do provide (and require) strong MFA — and only strong MFA. This is exactly the kind of data that most needs to be protected.
These identity frustrations aren't exciting. But they're real, and they're exactly the kind of problems we're working on at Oblique.


What we can learn from real-world authentication failures
Maya Kaczorowski — Wed, 15 Oct 2025 17:33:15 GMT
This blog post is a written version of a talk that our cofounder Maya gave at BSidesSeattle in April. You can also watch the recording and get the slides.
Authentication is one of the most critical identity controls we have. In some sense, it's both the first and last line of defense. If authentication fails, it means that attackers can bypass all other controls. And since identity is tied to all other systems, the compromise of even a single account could have a wide-reaching impact.
Although there are so many ways that authentication could go wrong — credential compromise, MFA and recovery weaknesses, session management failures, and poorly implemented protocols — there’s not necessarily that much variance in reality. In this post, we'll look at real-world authentication failures from the last five years. Studying these incidents is how our industry learns and improves, as long as we actually apply those lessons.
Credential compromise
Credential compromise is the most common type of authentication failure, whether that be password theft, credential stuffing, or a brute force attack. This could also occur via the compromise of an identity provider or password manager.
We know how to prevent most credential compromises: by using multi-factor authentication. Yet, these incidents remain shockingly common and impactful in reality. The real failure isn't "credential compromise," it's "no MFA” (let's call it what it is).
LAPSUS$ gains access to Okta’s support system
In January 2022, LAPSUS$ compromised an Okta third-party support engineer's laptop, using it to then access the customer support system. The breach only came to light months later when screenshots of Okta's own Okta instance with superuser access were posted on Twitter.
In all, the attacker had access to the individual’s laptop for 5 days, giving them access to 366 customers' Okta tenants, though seemingly no further damage was done.
Many of the authentication vulnerabilities we've seen in the past few years can be attributed to LAPSUS$ — and they definitely have their preferred playbook.
Another compromise of Okta’s support system
In 2023, an attacker again gained unauthorized access to Okta's customer support system, affecting 134 of their customers. The breach occurred when an employee saved their work credentials to their personal Chrome password manager, which was later compromised.
The attacker found that some support chats, logs, and files had valid session tokens, which they were able to use to compromise five of Okta’s customers, including Cloudflare, BeyondTrust, and 1Password.
Credential stuffing Snowflake customers
In early 2024, members of online crime-focused chat group The Com targeted Snowflake. Many of Snowflake’s accounts were only protected with a username and password (no second factor). UNC5537 purchased customers' credentials on the dark web and systematically took over these accounts.
Up to 165 Snowflake customers were compromised, with attackers stealing phone and text message records for 110 million AT&T customers and leaking 160 thousand Taylor Swift Eras tour barcodes from Ticketmaster. The attackers also extorted dozens of companies in exchange for not releasing their data, netting about $2 million in ransom payments.
In response, Snowflake made MFA generally available, requiring it for new accounts — and increased the minimum password length from eight to fourteen characters.
These credential compromises demonstrate that basic authentication still fails spectacularly. But what happens when organizations do implement MFA — and attackers find a way around it?
MFA and recovery weaknesses
When basic credential theft fails, attackers pivot to target the second factor directly. MFA and recovery weaknesses include MFA fatigue, implementation issues in recovery flows, and SIM swapping.
Social engineering to get access to Twitter's support tool
Twitter’s support team used an internal admin tool for common support workflows, such as suspending accounts or changing their recovery emails. Knowing that as many as 1500 Twitter employees had access to such a powerful tool, attackers scraped LinkedIn for Twitter employee profiles, then called them impersonating the IT department. The attackers directed their targets to a fake VPN login page and phished their 2FA code, gaining access to Twitter’s internal network, including the admin tool.
In this way, the attackers were able to gain access to about 130 prominent accounts, tweeting scam bitcoin messages from 45 of those accounts. This netted them around $120k in bitcoin payments before the messages were taken down.
MFA fatigue at Uber
To get into Uber, LAPSUS$ obtained VPN credentials for a contractor, then continuously bombarded that user with MFA prompts while simultaneously contacting them on WhatsApp posing as ‘tech support’. Eventually, the user approved a prompt, allowing the attack to log in. Once on Uber's VPN, the attacker had access to the contractors’s accounts on Google Workspace and Slack.
This was the same technique LAPSUS$ used against Microsoft, Nvidia, and Okta. MFA fatigue works because it only takes one approval — whether from exhaustion, confusion, or a simple misclick — to let attackers in.
Session management failures
Even with second factors in place, attackers adapt. Some have moved to bypassing authentication entirely, by stealing active sessions — through session hijacking, cookie theft, OAuth token theft, XSS, or CSRF. Once a legitimate user has authenticated and proven their identity, the attacker simply takes over that established session.
GitHub OAuth token theft
GitHub issues OAuth tokens for connecting with other systems like CI/CD pipelines. In April 2022, GitHub discovered attackers using stolen Heroku and Travis-CI OAuth tokens to access organizations’ data. Each token allowed the attackers to access the GitHub API as that user, enumerate their organizations and private repositories, then selectively clone repositories they were interested in.
The attackers then searched private repos for secrets. They found npm's AWS infrastructure API key, and proceeded to download all private npm package manifests and metadata, 100k npm users' hashed passwords, and private packages from two organizations.
OAuth tokens — which often provide broad, persistent access — have become increasingly attractive targets because organizations rarely review which apps employees have authorized or what scopes those apps have been granted.
Stealing CircleCI sessions and customer tokens
In January 2023, CircleCI was alerted to an issue when one of their customers noticed suspicious activity for a GitHub OAuth token that was stored in CircleCI. They discovered an attacker had compromised an engineer's laptop with malware, stealing an active 2FA-backed SSO session. The engineer had production access, which the attacker used to steal customer environment variables, tokens, and keys.
CircleCI issued a notice for its customers to rotate secrets they had stored in CircleCI, and then went a step further, coordinating with other infrastructure providers to invalidate and rotate OAuth tokens for GitHub, Bitbucket, and GitLab.
Authentication protocol failures
The last kind of authentication failures we’ll consider are authentication protocol weaknesses. These are rarely weaknesses in protocols, and more often stem from poor implementations, such as OIDC misconfigurations or missing JWT validation.
ProxyShell vulnerabilities in Microsoft Exchange
At Pwn2Own Vancouver 2021, researchers discovered vulnerabilities in Microsoft Exchange. By exploiting an "Explicit Login" feature used for direct access to a user's inbox, attackers could manipulate Exchange's URL normalization and gain unauthenticated access to Exchange Server internals. Chained with two other CVEs — collectively called ProxyShell — this enabled remote code execution on multiple Exchange Server versions, affecting tens of thousands of servers globally.
Microsoft quickly released patches, but about a year later, suspiciously similar attacks were detected. It turns out that the patches were insufficient: Exchange remained vulnerable to server-side request forgery, with two more CVEs — dubbed ProxyNotShell — allowing authenticated attackers to bypass two-factor authentication to again achieve RCE.
Lessons learned
So, what can we learn and apply from these real-world failures?
Implementing authentication
If you’re implementing authentication:
Check for common implementation mistakes. These can happen to anyone. Not even sophisticated organizations — like Microsoft, with critical authentication products like Active Directory, or Okta, an identity company — are immune.
Use the right algorithms. Double check anything that is particularly sensitive, or touches cryptography, like hashing or input normalization.
Implement session timeouts. Limit the impact of session hijacking by restricting how long a token or credential is valid for.
Prevent brute force attacks. Implement rate limits and log authentication attempts to avoid password spray or mass credential stuffing attacks.
Provide users with more than just basic auth. Implement SSO, MFA, and strong controls for admin or other sensitive actions.
Using authentication
So many of the issues we’ve seen in the last few years have very much followed a playbook — often LAPSUS$’s playbook. If there’s only one thing beyond the basics that you should really be doing, it’s rolling out strong phishing-resistant MFA, like hardware tokens or passkeys.
The basics still matter most:
Implement MFA. You should really do this!
Require the use of strong MFA. Use hardware or WebAuthn-based MFA.
Educate your users. Warn employees about attacks like MFA fatigue, and make sure they know how to report suspicious activity.
Set up monitoring. Alert on suspicious authentication events like MFA failures, password resets, and MFA enrollment for new devices.
Beyond MFA, you should also:
Review OAuth tokens in your environment. Check which integrations are authorized, what scopes they have, and when tokens were last rotated. Treat this as an ongoing process, not a one-time audit.
Prevent employees from using personal password managers. Provide them with a corporate password manager.
Pay attention to contractor access. Secure how non-employees, including those in support, access critical systems. This includes endpoint protection or browser controls to detect keylogging or infostealing malware.
Finally, assume compromise will happen and prepare accordingly. Design for defense in depth, implement logging and monitoring, and when a breach does occur, move quickly. Lastly, please share what you can — that's how the industry learns and improves.
Authentication failures continue to be a primary vector for major security incidents. Our best defense is learning from the past and preparing for what seems inevitable.


Internal security tools rarely get the UX they deserve
Maya Kaczorowski — Tue, 07 Oct 2025 16:18:43 GMT
Building internal tools is hard. Not necessarily technically hard, but it’s hard to build tools that people actually want to use. Security teams are good at creating access request portals, compliance dashboards, and exception workflows that technically work, but that are painful for our users — and then we wonder why these controls and processes get circumvented. When we build internal security tools, we consistently underestimate the time, investment, and ongoing maintenance needed to build something that people actually want to use.
We’re thinking about this constantly as we build access management tools at Oblique. We've experienced firsthand how challenging it is to create security tools that people actually embrace rather than reluctantly tolerate. An employee isn't necessarily thinking of the security implications when they share superadmin creds with a teammate who urgently needs to debug an issue, or when they download a sensitive financial report to a USB key to get printed at FedEx — they're just trying to get their work done when the approved path is too slow or unclear. And yes, I’ve done both of these things.
Whether you’re building because there isn’t a good enough solution on the market or because your environment is unique (or because you think it’s unique), the challenge remains the same. Before you commit to building — or deploying — a new security tool, step back and consider the full cost: you're committing to UX work, user support, integrations with every tool in your environment, and maintenance that will outlast the engineer who built it.
Security engineers aren’t the best product engineers
I’ll say the quiet part out loud: security engineers aren’t the best product engineers. (But that’s okay, because that’s not why you hired them!) Security teams now hire engineers who can build secure systems, not just analysts who review them. But engineering secure systems and designing intuitive interfaces are different skills — and most security engineers were hired for the former, not the latter.
Most security teams have limited engineering resources that need to support many teams: analysts, compliance, and IT. These engineers often focus on building core infrastructure and backend systems — with less experience in frontend and user experience. (This isn’t to say full stack snowflakes don’t exist, they’re just rare.)
When security teams do build tools, just like with any other development, the timeline estimation is completely off. We estimate the time required based on the core functionality we need, but forget about everything else: monitoring and logging, integrations with the IdP and SIEM, and testing. We struggle just to ensure coverage across internal systems — there's no time left for polish. UX improvements, documentation, and internal training all get cut.
Internal teams also don't generally get the full support you need to build a product: they don’t get product managers, designers, or docs writers. And if you do get allocated this help, you’re lucky to get a passionate volunteer. PMs working on internal tools tend to be evaluated against the same criteria as those shipping customer-facing products, putting them at a disadvantage for promotions and limiting career growth.
Usability matters for security
OK, everyone builds terrible internal tools — so why pick on security? Because usability actually matters in how effective your security program is. When our tools are too painful to use, people develop workarounds. An access request workflow that requires justifications that are never read and time-consuming approvals from multiple people teaches engineers to just share credentials instead.
A solution that's ‘good enough’ with unnecessary or unclear steps isn't actually good enough. It becomes an ongoing support burden: constant tickets for help because users can't figure out basic workflows, and requests for new functionality or integrations with new systems. This gets worse when the person who originally built the tool has moved on, and no one else understands how the black box works or how to modify it.
Security teams also deal with high-risk, high-stress workflows — do you really want someone to cause an outage, miss a critical piece of information, or accidentally send an alert because the UX wasn’t clear?
Choosing where to invest
When you do decide to build, invest properly from the start: think about UX, documentation, training or education, and exceptions. Explicitly hire for frontend and UX experience on the security team, and look for engineers with a product mindset or experience building for end users. Plan for integration work, ongoing maintenance, and support. Actually test your tool with your users and gather feedback. Track adoption, retention, and support tickets. If you're getting more support tickets than usage after launch, the tool isn't working. (Basically, act like it’s a real product — because to your users, it is!) If you don’t make that investment, it’ll all be a waste of engineering time.
And if you’re not willing or able to make that investment, realize you’re creating debt that you’ll still have to address later. Your engineering time is too valuable to waste on tools that people actively avoid using.


Modern access controls: takeaways on what actually works
Maya Kaczorowski — Wed, 24 Sep 2025 20:40:25 GMT
As we build access control policies into Oblique, it’s been interesting talking to IT and security teams to get examples of what they’d like to be able to implement. They’re shockingly… boring and unoriginal? Access management is one of those areas where organizations converge on similar approaches, because everyone is working with the same constraints. Even diverse organizations are generally protecting similar types of assets, dealing with similar compliance requirements, and working with similar tech stacks.
So, we wanted to document what IT and security teams have actually implemented today, in a new report on Modern access control policies, based on interviews with leaders from organizations with 125 to 5000+ employees. Here are the takeaways:
Policy ownership is shifting from security-only to being jointly owned by security, IT, and the business. While security teams historically defined policies and IT implemented them, organizations are moving toward shared ownership. More importantly, they're delegating authority to business teams, by involving them in defining requirements, setting policies, and handling approvals — rather than IT or security teams who don't have the context.
Controls should apply to data, not systems. The strictest controls should protect the most important assets: customer environments, customer data, and data within the scope of a compliance framework. But instead of focusing on protecting systems, teams are starting to think more holistically about data classification — even if that’s just distinguishing between systems that do and don’t have customer data — and protect resources based on the kinds of data they have.
More complex policy requirements are enforced when access is changed. Access requirements can be enforced both when access is used, and when access is changed. Zero trust policies focus on real-time context around users and devices when allowing access to a resource. But the more nuanced requirements kick in when access is granted, modified, or renewed. These decisions require business context from HR systems and validation of requirements that simply aren't available at the time of access.
Moving approvals from IT to managers improves speed initially, but human approvals remain a bottleneck. Delegating access decisions to managers notably improves the speed of an approval. But this initial unblock isn’t the end state. It’s often still too slow, and weakens security, since managers will approve any request to unblock their team. The real security and speed improvements come from asking those with context for approvals — app owners — and automating approvals entirely where requests are consistently approved.
Pre-approving specific groups for access is necessary when implementing approval workflows. To successfully reduce standing access, users need a way to regain that access validly without a cumbersome process. How can this ease of use be balanced with minimizing risk and cost? Pre-approve a set of users who can invoke access when they need it. This approach works well for common patterns, like engineers accessing production during on-call periods, or support staff accessing customer data with valid ticket numbers.
This report reflects what organizations are actually doing, not what they aspire to implement. Real-world organizations are, well, realistic. They need to balance security requirements while still keeping their organizations functional.
If you’re deciding what to implement in your organization, your peers have already figured out what works. Read the report, learn from their experiences, and apply the same patterns to your organization.


Delegate authority to those with context
Maya Kaczorowski — Wed, 17 Sep 2025 18:11:58 GMT
Early in my career, I thought the hard problems in security were the technical ones. Figure out the right architecture, pick a modern cipher suite, write it all in a memory-safe language — that's the important stuff, right? I grew up. People and process problems are always the hardest to solve, especially at scale. And access management is a perfect example of this.
IT and security teams don't want to be gatekeepers — they want to enable the business to move quickly. But if every access change, every application exception, every device approval needs to go through them... then they are gatekeepers. As it relates to access, the problem is that IT often lacks the context to make the right decisions quickly: who is Bob and whose app is it anyway? Meanwhile, business teams have the context but are stuck filing tickets. The solution isn't more people reviewing tickets — it's giving the right people the ability to make access decisions directly.
We've already started delegating security
Engineering figured this out years ago: we have CODEOWNERS for repositories, on-call rotations for specific services, and now developer portals to help consolidate all of that information in one place. Security has learned this lesson too: when we find a security issue, we need to know who’s responsible for it, so that we can ask them to fix it. Security teams face the same ownership question everywhere: which team owns this vulnerable dependency? This unpatched server? This open firewall rule?
We've had the most success with code. Finding bugs centrally and assigning them to dev teams was a disaster, so we "shifted left" and made engineering teams own their vulnerabilities. It's not perfect, but it works better: developers know their code and can prioritize and fix the vulnerabilities they find. We’ve already been delegating the work of fixing security issues, but we’re just starting to give these teams the control over what they prioritize.
Corporate security is also heading in this direction. A few years ago, Kolide innovated in endpoint management by delegating responsibility to end users to install security updates or turn on disk encryption. Instead of IT managing MDM exceptions through tickets, users got direct notifications about security issues on their devices, and were often able to resolve them themselves, without ever opening a ticket. This kind of delegation is possible because it's clear who owns issues for a given laptop: the person signed into it.
More and more tools are embracing this model: Workshop lets teams collectively attest that specific binaries should be allowlisted on their devices, rather than waiting for centralized approval. The team that actually needs Android development tools is better positioned to justify that need than someone in IT who's never built an Android app.
The logical next step is applying this same principle to access management — and some organizations already are.
Distributed access control isn't new
The reality is that access management is already distributed, whether you know it or not. Most SaaS apps are already managed in a decentralized way. The marketing team set up the CMS, so someone on the marketing team (you're not sure who) controls access and invites new colleagues. Some SaaS tools like Slack and Figma let you join automatically if you're part of the same domain, to make it easier for teams to work together (and to grow that per-seat revenue). This happens in smaller organizations before you centralize control with an SSO provider, but even then, there's always shadow IT or apps that don't support SSO.
Delegating authority in access management isn't a new idea. This is just discretionary access control: every resource has an owner, and that owner decides who gets access. It's exactly how sharing a Google doc works.
When we started building Oblique, we found that many tech-forward organizations had already built internal access management systems around this principle. Every single one had ownership baked in — they needed to know who was responsible for each app, service, resource, or group so that they knew who to bother when someone needed access or when compliance required an access review.
App or service owners handle the actual access decisions, including inviting, approving, or reviewing access for what they "own." They've been delegated both the responsibility and the decision-making authority to decide who actually needs access. This isn’t a title, but a responsibility. And this isn't about hierarchy, but about proximity to the resource. A senior engineer who built the service is a better choice than a VP who's never used it. Importantly, you need multiple owners — if your single owner goes on parental leave or quits, you're back to having IT make decisions about things they've never touched. Not only does IT lack context on what this resource is, they have even less context than usual since they haven't been involved in the recent access decisions.
Delegation is how security teams scale
Successful delegation means giving teams authority, not just responsibility. Let the head of Sales decide who gets CRM access: they understand what closes deals, know the cost of additional seats, and have both the business context and financial incentive to make good decisions.
Delegating isn’t easy. It’s scary: what if something goes wrong? But just like a new manager learns to give away more and more control — in a safe environment — security teams are learning to give away more and more responsibility. This shouldn’t be done blindly, but with guardrails to prevent disasters, and accepting that there will be exceptions.
Acknowledging that security can't do everything is part of growing up as an organization scales. Just like in application security or other parts of corporate security, successful delegation in access management means identifying owners, defining their responsibilities — and most importantly, giving them the authority to make decisions that work for them.


Stop trying to make git happen
Maya Kaczorowski — Wed, 10 Sep 2025 16:55:24 GMT
In the past few years, engineers have really come to embrace “as code” or GitOps workflows. And what’s not to love: you can stay in an editor you’re familiar with, and preview, test, and deploy changes automatically. This has been popularized for many things developers need to manage — we’ve seen infrastructure as code, config as code, and policy as code. Security teams have gravitated toward these workflows too.
It’s no surprise security teams love these workflows too: you get version control, reviews, rollbacks, and audit logs… for free! If the only user of your tool is the security team, this might be reasonable. But if your internal tool needs to be used by anyone who isn't an engineer, it needs to be usable for everyone — and as code workflows aren’t for everyone.
Not everyone can use git
As code workflows become a barrier as soon as someone outside the engineering org needs to use an as code tool as a part of their job. Do you think everyone outside of engineering — in sales, marketing, support, product, HR, or ops…
… has a GitHub account?
… knows Markdown?
… can open a PR?
… can set up a dev environment, even if it’s a devcontainer or codespace?
No.
I worked at GitHub, and I still don’t feel comfortable using git most of the time, even by referencing excellent documentation. I refuse to use the git CLI. And I would rather throw my computer out the window than figure out how to rebase my branch. (I shouldn’t litter my neighbor’s yard, so what I often do is just start a new PR.) I’m a more technical user — and if I can’t use git, it’s just not reasonable to ask your employees to.
Forcing managers to update permissions in git, or HR to create new accounts in a YAML template can’t be solved with extensive training — realistically, you’re looking at constant support requests. They’ll come to you every time they need help with that task, because you haven’t given them the tools they need to do it themselves.
You might even try to build a UI on top of your as code system, which means keeping context in sync between code and a user interface. Now you’re juggling UI changes and git branches, generating PRs through bots for basic edits, and still don’t have a good flow when a reviewer wants to tweak the generated code. Congrats! You’re now using git as a bad relational database.
Low-code tools exist for a reason
We have lots of internal tools that less technical users need to use — or build. We’ve seen an explosion in popularity in low-code or no-code portals for these tools (and now, LLM-enabled workflows). People who want workflow automation don’t necessarily code, and people who need to use those workflows also don’t necessarily code.
AI has certainly made this easier, but not necessarily better. Claude means that I can now generate a PR against an as code system for an access request, but is a critical everyday workflow really what you want me to be vibing?
So why do we keep building internal tools that require git? Often, because as code provides properties we want, like version control and review workflows. But optimizing only for what you need as a builder — and ignoring what your user (which could be anyone else in the organization) needs — is how you end up with a tool that no one uses.
When you’re building internal tools for everyone in the org, they need to work for everyone in the org. User experience matters more than developer convenience.
"As code" doesn't literally have to mean as code
When we started building Oblique, we started with as code definitions for access controls. That’s what IT and security teams were asking for, and what so many of them have built internally. But the more we talked to these teams, the more we realized what they wanted were the tools to preview access changes and ensure the right review requirements are met, in order to be able to safely make access changes. They wanted the benefits of as code workflows.
“As code” doesn’t literally have to mean as code. You can get the same benefits by building a thoughtful system that takes your change management requirements into account, without making someone learn git.
As code workflows make sense for systems that engineering owns, like infrastructure. But for systems that everyone in an org needs to use, like managing corporate access, requesting policy exceptions, or reviewing vendors? Build those tools with the people who actually use them in mind.


Why your RBAC roles aren't actually roles
Maya Kaczorowski — Tue, 02 Sep 2025 17:06:40 GMT
Most security teams understand Role-Based Access Control (RBAC) basics: map permissions to roles, assign users to roles, and you can control access. Simple enough. Well, except that every tool you use implements RBAC slightly differently, so you do need to think about it: who should be our GitHub CI/CD Admin? What’s the difference between a Vercel Member, Developer, and Contributor?
But there's a fundamental confusion that makes RBAC implementations difficult to manage: we keep using job titles as access roles. You should never encounter a "Growth Hacker" or "Chief Happiness Officer" in your access control system, even though they might be in your org chart. Your job title makes a bad RBAC role.
Roles describe what you do, not who you are
A role in RBAC should represent what someone actually does in your environment. Your job title, on the other hand, is about reporting structure, compensation bands, and career progression. It's your position in the corporate hierarchy, not your function.
Roles like "Customer Support" or "IT Admin" might make sense in both contexts, because they not only describe specific job functions, but are also tightly coupled to typical tasks someone in that position needs to accomplish. These roles are the exception rather than the norm.
Take a "Product Manager" instead. A Product Manager working on billing needs access to Stripe and support tickets, but a Product Manager working on Android needs access to app builds and bug reports. Same title, completely different functional needs. If you create a single "Product Manager" role, you're forced to either over-privilege everyone or create dozens of variations like "Product Manager - Billing" and "Product Manager - Android." What you actually want is a role that describes the billing team or the Android team.
What makes this worse is that job titles change constantly — should changing your job title in Workday to “AI Engineer” really change access? — and what those jobs actually entail evolves even faster. In today's flexible organizations, people wear multiple hats that don't fit neatly into traditional role definitions. (Seriously, what is a "GtM Engineer"?)
How we ended up in this mess
When organizations first adopt RBAC, they might go through a "role mapping" exercise to consolidate permissions into common workflows. Too often, IT leads this process without sufficient business context. It's easy to map existing permissions to a role that exists in an org chart, like "Marketing Manager," without asking if that's really the right access control boundary — or worse, a “Vendor” or “Partner”.
Being too generic doesn't help either. An "Admin" role doesn't actually mean anything. And "Data Reader" could represent 10 different job titles with completely different needs. You can't create meaningful roles without understanding what work actually looks like.
This gap has become glaringly obvious in the past few months as we rush to deploy AI agents: just like humans need appropriate roles, agents need appropriate OAuth scopes. We're struggling with the same “role explosion” problem: it’s unrealistic to create hundreds of OAuth scopes, and granting broad access is risky. We never learned to think functionally about access control for humans, so we're making the same conceptual mistakes with agents.
Making RBAC work for you
So, how should you think about adopting RBAC?
First, work with business teams to define roles. Every organization is different, and there's no one-size-fits-all approach. Ask business teams: who does the same tasks? Who works together on this specific type of work? People with completely different job titles might end up with identical access to systems.
Good functional roles sound boring and obvious: “Blog Publisher”, “Invoice Processor”, or “Customer Data Viewer”. Many people across different teams might need to publish blog posts, so they all need access to the publishing platform. If your role name requires a hyphen or sounds like it belongs on a LinkedIn influencer's bio, you're probably doing it wrong. Keep definitions simple enough that non-technical stakeholders can understand — and request access to — them. Taking the time to define roles that are clear and have sufficient context, as this will save you time later.
Use fewer, broader roles rather than overfitting to every scenario. When you have hundreds of roles, you've created a system that nobody understands. Role mapping turns into “role mining” or even “role engineering”. Role sprawl isn't just annoying — it's impossible to debug when something goes wrong if you can't tell why roles exist, who should have them, or how two similar-sounding roles differ.
Start by only defining roles that you actually need to restrict. Split out the riskiest permissions into roles that need to be more tightly controlled, and keep a reasonable base set of permissions for everyone else. You can start with basic permission levels that apply to any resource, like Reader, Writer, and Admin. These boring old roles exist because they actually work — your cloud provider didn't pick them by accident. Don’t define roles for the sake of it, but as you have a need to enable new users and restrict sensitive permissions. Accept some over-privilege to avoid the administrative hell of managing hundreds of hyper-specific roles. You'll have exceptions regardless, so spend your time building a decent exception process instead of trying to create the perfect role taxonomy.
Plan for ongoing maintenance. This is the part that often gets left by the wayside. Roles need regular review and updates — you can even do this at the same time as your user access reviews! Instead of just checking if Bob should still have the Admin role, also review what the Admin role can actually do and whether that still makes sense. Roles have a nasty habit of scope creep, quietly accumulating permissions over time as teams request access to new tools and systems. Check periodically that your "Viewer" role can't actually modify data. Audit role capabilities, not just role assignments.
Mix and match your access models
Real organizations are complex, people wear multiple hats, and business needs change faster than security can keep up. The issue isn’t RBAC, it’s how organizations implement RBAC.
Most organizations don’t implement a “pure” access control model. Instead, they blend RBAC for stable business functions, ABAC for context-dependent decisions, and other approaches as needed. Rather than forcing everything into roles, let attributes be attributes — someone's department is useful as an attribute for access decisions, not as another role to manage.
Keep RBAC functional
We struggle to maintain access controls because we've designed them around what people are called rather than what they actually do. Job functions matter for access control, but they're just one piece of the puzzle. Where you're working from, what time it is, and who you're collaborating with often matter more than your title.
At the end of the day, RBAC is an imperfect tool. Design a sane and maintainable set of groups, focus your security efforts on the riskiest permissions, and build processes to handle the inevitable exceptions and changes coming your way.


Comms groups inevitably become access groups
Maya Kaczorowski — Thu, 28 Aug 2025 17:09:40 GMT
IT teams create two types of groups that seem logical in theory: communication groups and access groups. "Comms groups" are meant for project updates and discussions, and so are easier to join. "Access groups" are meant for actual system permissions and are much more tightly controlled — managed by IT, and with approval requirements set by security.
The theory sounds reasonable: keep the friction low for people who just need to stay informed, but maintain strict controls for actual access.
This doesn't work in practice. Comms groups always become access groups. It's not a matter of if, but when.
How groups actually get used
This happens because in many platforms, the same group management systems that handle communication also control access. In GCP, your Google Groups directly become your IAM groups. In Slack, when you share a document with a channel, you're granting access to everyone in that channel. A comms group for an offsite ends up being used for access to a new tool built during the event. A social activities group for the Boston office becomes the access control for the newly installed bike racks. IT teams try to prevent this — usually by creating naming conventions like “Project-X-COMMS-ONLY”. Just like your file naming convention — you know, the one where “Filename_FINAL_v3” gets superseded by “Filename_FINAL_v4_ACTUALLY_FINAL” — it's a beautiful lie that doesn't work.
Users will follow the path of least resistance. When you need to grant access to a system, and everyone who needs access is already right there in that group, you use it… even if it just happens to be the COMMS-ONLY group. (It’s rare that the reverse happens.)
Why is the comms group often a better fit for access? Comms groups map to how people actually work, and often, access groups don't (but they should). Access groups tend to be tightly managed to meet specific compliance or security requirements. As users often need to request access to these, they become bureaucratic obstacles that get in the way of actual work.
Comms groups, on the other hand, naturally follow project and team lines — they contain the people you need input from or need to keep informed. IT teams typically want these groups to be easy to join since they enable business operations and seem low-risk (even when the conversations themselves involve highly sensitive information). Our access controls end up mirroring our communication patterns, whether we plan for it or not.
The gap between groups and permissions
Part of the reason IT teams try to create this separation is that we fundamentally lack visibility into what we're actually granting when we add someone to a group. The system managing groups (often your IdP) isn’t the same system where your permissions live (often an application).
This is where security breaks down: when you add a user to a group, you're making an access decision blind to the actual access being granted. You can't see what downstream permissions a group membership grants — access to customer data, financial data, or other sensitive resources. (Look at Google Groups’ newly added security groups, which perpetuate this naming separation while offering only limited restrictions. It's too little, too late — and too confusing to solve the real problem.)
IT teams try to separate comms and access groups to reduce what they need to manage, control, and audit, but sensitive access inevitably creeps into places it doesn't belong.
Design for reality
So what can you do? Well, stop fighting this pattern, and accept the inevitable. Instead of pretending comms and access groups will remain separate, create unified project- or team-based groups from the beginning. Make it explicit that joining the project group means both receiving comms and getting relevant access.
The goal isn't to make it even harder for users to get access — they already find it plenty frustrating. Acknowledge that comms and access groups are one and the same, and design proper controls from the beginning.


Injection-proof SQL builders in Go
Eric Chiang — Mon, 18 Aug 2025 15:00:00 GMT
A Go product that uses SQL will inevitably implement some higher level logic on top of database/sql. There are just too many cases where a single string with a fixed set of arguments isn’t flexible enough. Using different database flavors for dev and prod which take different placeholders ("?" vs "$1"). Inserting multiple rows in a single statement. Performing the same query with different WHERE conditions.
While builders are often necessary for development, they’re also absolutely terrifying for security.
Sure, Go has built-in parameterized values for input variables, but what if we’re trying to specify column, row, or table names? Most packages will happily accept arbitrary input in these fields and run it directly against your database.
// Runnable example: https://go.dev/play/p/bGiCWp6xk-z
package main

import (
	"fmt"

	"github.com/huandu/go-sqlbuilder"
)

func main() {
	userInput := `1;
		DROP TABLE students;
		SELECT (id, name) FROM demo.user WHERE status`
	where := `status = ` + userInput
	// ...
	sql := sqlbuilder.Select("id", "name").From("demo.user").
		Where(where).
		String()
	fmt.Println(sql)
}
When building Oblique, we weren’t thrilled with the idea of accidentally introducing a 90s vulnerability into a security product built in 2025. If a customer trusts Oblique to manage authorization in their environment, we should do more than hope our new backend hire doesn’t misuse a Go API.
A better way
There turns out to be a clever trick with the Go type system to ensure an argument is free from dynamic input. That way, we can constrain the builder’s inputs rather than sanitize or detect after the fact.
Consider the following package:
package say

import "fmt"

type myString string

func Hello(name myString) {
	fmt.Printf("Hello, %s!\n", name)
}
Because myString is private, there’s no way for an external package to create a variable of that type.
This should make it impossible for another package to call say.Hello, except for one notable exception. Go is relatively strict on mixing types. You can’t add an int to a float64 or even an int to an int64. To compensate, Go constants allow programs to define untyped values whose type is inferred when they’re used in some context that requires one. That’s why you can do something like the following:
const thirtyDays = 30*24*time.Hour
Rather than explicitly typing every number:
const thirtyDays = time.Duration(30)*time.Duration(24)*time.Hour
The constants 30 and 24 are coerced to a time.Duration by being multiplied by time.Hour.
Let’s go back to our earlier example. While another package can’t create a variable with the myString type, it is possible to pass a constant! This will get typed as myString simply by being used as an argument value.
// Runnable example: https://go.dev/play/p/_0NyeiTp-M8
func main() {
	say.Hello("Eric") // Works even though the argument is a private type.
}
Constants and builders
We can use this observation to force callers to only pass constants to our APIs, which by definition will never be dependent on dynamic input. You can’t construct a constant using fmt.Sprintf or other string concatenation that depends on a live variable.
Here’s a full example builder that uses private string types for column, row, and table names:
package sqlb

import "strings"

// Private string types that can’t be referenced by other packages.
type col string
type row string
type table string

// Helpers to construct types dynamically, while still requiring
// constant strings as arguments.
func Row(val row) row        { return val }
func Rows(vals ...row) []row { return vals }
func Table(val table) table  { return val }

type SelectBuilder struct {
	rows    []row
	from    table
	whereEq *whereEq
}

func Select(rows ...row) *SelectBuilder {
	return &SelectBuilder{rows: rows}
}

func (b *SelectBuilder) From(table table) *SelectBuilder {
	b.from = table
	return b
}

type whereEq struct {
	col col
	val any
}

func (b *SelectBuilder) WhereEq(col col, val any) *SelectBuilder {
	b.whereEq = &whereEq{col, val}
	return b
}

func (s *SelectBuilder) String() string {
	b := &strings.Builder{}
	b.WriteString("SELECT (")
	for i, row := range s.rows {
		if i != 0 {
			b.WriteString(", ")
		}
		b.WriteString(string(row))
	}
	b.WriteString(") FROM ")
	b.WriteString(string(s.from))

	if s.whereEq != nil {
		b.WriteString(" WHERE ")
		b.WriteString(string(s.whereEq.col))
		b.WriteString(" = ?")
	}
	return b.String()
}
With this package, you can still write the basic builder logic you’d expect:
// Runnable example: https://go.dev/play/p/Nauyr8LNdUr
var rows = sqlb.Rows("id", "name")

func main() {
	sql := sqlb.Select(rows...).
		From("demo.user").
		WhereEq("status", 1).
		String()
	fmt.Println(sql)
}
But, if you ever accidentally depend on a string variable, the program refuses to compile!
var rows = sqlb.Rows("id", "name")

func main() {
	userInput := ` = 1; DROP TABLE students;`
	whereCol := `status` + userInput
	// ...
	sql := sqlb.Select(rows...).
		From("demo.user").
		WhereEq(whereCol, 1).
		String()
	fmt.Println(sql)
}

// cannot use whereCol (variable of type string) as sqlb.col value in argument to sqlb.Select(rows...).From("demo.user").WhereEq
Helpers like Row(val row) row still allow other packages to dynamically construct the set of rows to select, but guarantee their values trace back to constants and aren’t dependent on user controlled values.
Footguns
Smart engineers at prominent companies accidentally write vulnerabilities like this all the time. As a codebase gets bigger, it’s just not possible to depend on human review as the security check. Nor is this restricted to SQL. Similar issues with JWT libraries and archive unpacking are routine when APIs make it easy to accidentally do the wrong thing.
If you are building a package with security implications, it should be as hard as possible (if not impossible) for users to do something insecure. For SQL builders, what’s better than insecure code not compiling at all?


The evolution of authentication, from passwords to passkeys
Maya Kaczorowski — Wed, 13 Aug 2025 15:00:00 GMT
This blog post is a written version of a talk that our cofounder Maya gave at BSidesSLC in April. You can also watch the recording and get the slides.
Authentication has always been defined by the tension between security and usability. For years, making authentication more secure meant making it harder to use — and making it easier to use meant introducing new security holes. Security innovations used to start in enterprise and trickle down to consumer applications. But now, that flow has reversed, with consumer authentication improvements like TouchID making their way into enterprise environments.
This push and pull between security and usability has been productive as both sides of the equation have improved over time. Increasingly, the best authentication controls we have are the most usable ones.
One thing hasn’t changed: we still need to verify that users are who they claim to be. But how we do that verification has changed: authentication has moved from simple passwords to complex passwords, from islands of identity to federated systems, and from single-factor to multi-factor authentication and passwordless experiences.
Basic authentication: usernames and passwords
Passwords are relatively new. In 1961, Fernando Corbató at MIT created passwords for time-sharing systems where multiple users needed different access levels and private files. This ensured users were only authorized to access their files and use their assigned time on the shared system.
Just a few years later, a doctoral student who wanted more time on that system realized that he could print out the list of all of the passwords, and login as someone else. Turns out the passwords were just stored in plaintext.
To protect against this password theft, Unix introduced password hashing in the 1970s. By using a one-way function (a hash), the system can store enough information to verify that users are presenting the right passwords, without storing, and therefore risk leaking, the passwords themselves.
As the number of systems that a user might need to log into increased, they had a corresponding need for more passwords, and so frequently a user would reuse the same password to authenticate to multiple systems. However, if a user reused a password in multiple places, and any one of those instances was compromised — for example, if the system didn’t actually hash and store passwords properly — then that password could be tested against another system to see if it would give access. Attackers could then use these compromised credentials in targeted attacks against specific users (credential stuffing attacks) or feed them into broader password cracking efforts to expand dictionaries of common passwords and their corresponding hashes.
To make their passwords easier to remember, users started using variations on the same passwords. For example, rather than using the 27th most common password hunter, they might instead use the password hunter2.
Password complexity requirements
To prevent weak passwords like password123, organizations introduced password complexity requirements. Rather than letting a user set any password, users’ passwords had to meet specific requirements in order to make them harder to crack.
When NIST SP 800-63 was first published in 2004, it suggested minimum requirements for password complexity, including irregular capitalization, special characters, and at least one numeral. Although it’s mathematically true that passwords using a variety of characters greater than just lowercase letters should be harder to crack, in reality, many passwords that humans create to meet these requirements end up becoming a predictable ordering of capital letters, lowercase letters, numbers, and symbols, like Password123! or leetspeak like P@ssw0rd!. Meeting specific password requirements can be infuriating.
Password complexity requirements didn’t improve security — they made it worse. Users couldn’t remember complex passwords, so they’d reset them more frequently. And, there often were additional requirements around regular password rotation. An easy-to-remember password is not secure, but a secure password is hard to remember, harder to type, and so less usable. It’s an unfortunate tradeoff.
The current industry guidelines instead focus on password entropy — which is achievable with a long password, like correcthorsebatterystaple — and discourage specific complexity rules and rotation of non-compromised credentials. Security doesn’t have to be completely at odds with usability: rather than a perfectly random password, a fairly random but pronounceable password is much more usable and still more secure.
Password storage has also improved: it’s easier than ever to properly salt and hash passwords. argon2, as the winner of the Password Hashing Competition, has taken over from bcrypt — not only is it fast, it also lets the implementer trade off speed and memory usage with requirements around side-channel and cracking attacks.
But by moving to long, high-entropy passwords that are unique for each system, this created yet another issue: how can users possibly remember all of these passwords?
Password managers
Password managers helped users manage their growing collections of credentials — 168 passwords on average. Available as both standalone apps like 1Password and built directly into platforms like Google Chrome and iOS. Password managers have become mainstream security advice, even superseding VPNs for family recommendations.
Password managers also gave IT teams visibility into password practices for the first time. Organizations could finally enforce password policies and see whether users were following them. But password manager adoption remained inconsistent, and the core problem persisted: too many systems requiring separate credentials.
Enterprise identity federation
Identity federation emerged because organizations hit two walls. Users couldn’t manage dozens of passwords, and IT teams couldn’t track access across multiple systems. Centralized identity management solved both problems: users get one set of credentials to remember, and IT gets a single place to manage access.
Microsoft Active Directory (AD), launched in 1999, was the first real solution to enterprise identity sprawl. Instead of maintaining separate user accounts for every server, printer, and application, IT teams could manage everything from one directory. AD became the “single source of truth” for who had access to what within the corporate network. AD implemented Kerberos as its authentication protocol, including Kerberos’ delegation and impersonation capabilities, which allowed services to act on behalf of users within the enterprise network.
However, AD only worked on the local network. SAML, introduced in 2001, tried to solve this by letting identity providers send authentication statements and attributes across domain boundaries, extending identity beyond solely AD-based systems.
When cloud services emerged in the mid-2000s, identity needed to extend even further. Hosted identity providers like Ping Identity (founded in 2002) and Okta (founded in 2009) were introduced to bridge the gap: to offer a way to federate on-premises identities into cloud SaaS applications. They weren’t replacing AD, but extending it to work with SaaS applications. This was the beginning of modern identity federation.
OAuth, introduced in 2007, brought delegation to web applications and third-party services. Instead of sharing passwords, users could grant specific permissions to external applications. Later, OpenID Connect, standardized in 2014, extended OAuth to handle authentication in addition to authorization. Together, these created a comprehensive identity stack:
Centralized identity stores (AD, LDAP) for centralized user, group, and access management
Federation protocols (SAML) to connect across domain boundaries
Cloud identity providers (Okta, Ping) to extend directories to the cloud
Authorization frameworks (OAuth) and authentication extensions (OpenID Connect) to authenticate delegated authority
The early 2000s Microsoft ecosystem was the golden age of enterprise identity — everything integrated smoothly, letting you authenticate the same way to your desktop, network, and printer.
Social login
Consumer identity saw similar consolidation. Social login emerged with Facebook Connect (2008) and Sign in with Google (2011), letting users use existing accounts to federate their personal consumer identities, rather than needing to create new accounts for every site they visited.
Social login succeeded because it solved the same problems enterprises were facing: reducing password fatigue while shifting authentication burden to providers with dedicated security teams. The time-consuming and risky processes for password reset and account recovery could instead be handled by providers with significantly more resources and experience.
Multi-factor authentication
Organizations moved from a single password for authentication, to two-factor authentication (2FA), to multi-factor authentication (MFA).
MFA means that just a password isn’t enough — instead, to verify a user, multiple verification types are needed. These are typically described as: “something you know” (like a password), “something you have” (like a device), and “something you are” (using biometrics). The idea is that merely a compromised password alone shouldn’t grant access — attackers would need to compromise multiple factors.
Many kinds of factors have existed over the years:
RSA SecurID tokens (1980s): The original “something you have”: physical devices on a keychain that generated time-based codes. Effective but expensive and annoying to carry — plus, the batteries died.
SMS codes (mid-2000s): Convenient, and what is still probably the most widely used second factor today. Though vulnerable to SIM swapping, SMS codes were crucial for making MFA mainstream and accepted. (Plus, they gave us all a legitimate reason to keep our phones within arm’s reach at all times!)
Authenticator apps (2010s): Google Authenticator (2010), Duo (2010), Authy (2011) improved security with TOTP codes generated on the user’s device.
Push notifications (mid-2010s): Instead of generating codes, what if those apps instead sent push notifications? Much more convenient, but these caused “notification fatigue” — users would approve notifications without thinking, even when they hadn’t actually tried to log in. This made phishing attacks easier, not harder.
Hardware security keys (mid-2010s): YubiKeys and other FIDO U2F hardware tokens generate unique codes when touched, providing proof of presence without requiring users to type anything. This allows for much longer codes, improving security without requiring more effort from the user. This became best practice when Google published a case study that showed that implementing hardware keys effectively eliminated successful phishing attacks in their environment.
Platform authenticators (mid-2010s): TPMs in devices enabled hardware-bound key material. Combined with biometrics like TouchID (2013), Windows Hello (2015), and FaceID (2017), these verify multiple factors: that both the user has the device, and the user is the person they say they are.
The biggest barrier to MFA adoption isn’t technical complexity — it’s that many applications still don’t support MFA. Just like with SSO, you’ll find apps that don’t offer MFA at all, or charge extra for it as an “enterprise” feature.
Passwordless
Passwordless authentication takes MFA to its logical conclusion: eliminating passwords entirely. Instead of adding secure factors to passwords, make the secure factor the only factor.
This first started with magic links (mid-2010s), which allowed for email-based authentication without passwords by sending a user either a link or a code via email. Magic links were especially popular for social logins, where forgetting your password meant that you could still access your account via email. However, these links are phishable, and allow for account takeover if your email is compromised.
The passwordless movement took off as biometrics became possible, then normal, and eventually, mainstream. Device biometrics (mid-2010s) allowed for platform authenticators like TouchID, Windows Hello, and FaceID to auth using a user’s biometrics — but instead of using these for second factors, using these to secure primary factors.
The WebAuthn standard launched in 2019, specifying how to implement FIDO2 for web APIs. By standardizing passwordless authentication for browsers, WebAuthn enabled phishing-resistant credentials across the web.
WebAuthn succeeded because it achieved the seemingly impossible: improving both security and user experience simultaneously. Hardware-bound credentials resist phishing since they’re tied to both physical devices and specific services. When FaceID replaced typing passwords on mobile devices — or entering a 2FA code, or selecting from a password manager — users adopted it because it was faster and easier, not because of its security benefits.
Passkeys
Passkeys were supposed to make WebAuthn mainstream. The rollout has been messy. Apple, Google, and Microsoft promised a better user experience in 2022: biometric-based FIDO credentials that work across platforms and devices. This would also improve security, as these phishing-resistant credentials are bound to specific origins.
Instead, passkeys created vendor lock-in and user confusion. Passkeys have been confusing at best, due to inconsistent experiences across providers. Cross-device syncing only works within ecosystems: Apple devices to Apple devices, and Chrome sessions to Chrome sessions. Users with multiple platforms face competing password managers during every signup or login, uncertainty about where their passkeys are stored, and unclear account recovery processes. And, passkeys still have limited support.
The industry was so excited about WebAuthn and passkeys (rightly so) that we bungled the user experience. Biometric unlocking for password managers and direct passkey authentication compete for attention during the same workflows. Despite being more secure and theoretically easier to use, passkeys aren’t something most of us would recommend to our families. The industry tried to create a better user experience and made things more confusing instead.
That said, passkeys do represent meaningful progress: moving from something users know (passwords) to something users both are and have (devices with biometrics).
Looking forward
The evolution continues beyond just login. Authentication and authorization are blurring together. As authentication becomes more continuous and adaptive, systems now verify both identity and permissions for every action, not just at login. This means usability challenges in authentication increasingly apply to authorization too.
The tension between security and usability that defined authentication’s early history is finally resolving. The best authentication controls — hardware security keys, TouchID, well-implemented SSO — are both more secure and more usable than what they replaced.
This matters for security teams beyond just authentication. Instead of assuming that better security means more user friction, look for solutions that improve both. The security controls that actually work are the ones people want to use.


Authenticating GitHub Actions without API keys
Eric Chiang — Thu, 31 Jul 2025 15:00:00 GMT
At Oblique, we’re building a modern service to define and manage authorization in your corporate environment, and an early “modern” decision we made was to be API-first. Everything a user sees in our UI should integrate seamlessly with RPC clients, configuration-as-code, and MCP clients.
This naturally leads into the question of how we authenticate those clients. Sure, we can mint a long-lived API key and allow-list your CI/CD system’s ~5000 CIDR ranges. But asking a user to copy and paste key material with a big “Keep this secret!” warning isn’t exactly what I’d describe as “seamless.”
Instead, let’s talk about workload identity and GitHub Action’s OpenID Connect support.
OpenID Connect
OpenID Connect is a protocol on top of OAuth 2.0 that allows third-party services to determine your email, account ID, and display name using standard fields (rather than every IdP implementing its own custom set of APIs). If I’m trying to figure out your email during an OAuth flow and I’m using OpenID Connect, the same code I write to log you into Google now works with Microsoft Entra, Keycloak, Dex, and Okta. Aren’t standards great?
At a low-level, the OpenID Connect token response includes a signed JWT from the IdP with pre-defined fields. Here’s an example payload from Google’s docs:
{
  "iss": "https://accounts.google.com",
  "azp": "1234987819200.apps.googleusercontent.com",
  "aud": "1234987819200.apps.googleusercontent.com",
  "sub": "10769150350006150715113082367",
  "at_hash": "HK6E_P6Dh8Y93mRNtsDB1Q",
  "hd": "example.com",
  "email": "jsmith@example.com",
  "email_verified": "true",
  "iat": 1353601026,
  "exp": 1353604926,
  "nonce": "0394852-3190485-2490358"
}
Now, if you give a developer a JWT signed by Google that says “this is eric@oblique.security,” they’re going to (ab)use them. Over the years, systems started accepting ID Tokens as a primary credential outside of OAuth2.0, and today you can authenticate directly to systems like Kubernetes, Vault, and AWS by presenting the ID Token itself from a command line or other non-browser system.
GitHub Actions credentials
GitHub Actions can request short-lived ID Tokens through an internal API exposed to the action. That token’s payload contains metadata signed by GitHub about the runtime environment and what triggered the run. Here’s an example payload with some fields omitted for brevity:
{
  "actor": "ericchiang",
  "actor_id": "2342749",
  "aud": "oblique.security",
  "exp": 1753783724,
  "iat": 1753762124,
  "iss": "https://token.actions.githubusercontent.com",
  "ref": "refs/heads/bash-script",
  "ref_protected": "false",
  "ref_type": "branch",
  "repository": "ericchiang/github-actions-oidc-example",
  "runner_environment": "github-hosted",
  "sub": "repo:ericchiang/github-actions-oidc-example:ref:refs/heads/bash-script"
}
For whatever reason, GitHub chose to implement Action identity as a single issuer for all workloads using global signing keys. This means that any Action can mint a valid signature for any system that trusts GitHub Actions. Yes, my dinky test repo can authenticate to your Kubernetes cluster. It hopefully isn’t authorized to do anything, but it’s up to the cluster to validate the action.
If the system that’s receiving the tokens doesn’t support custom field validation, you’re forced to pattern match the subject (“sub”) field, which follows the form “repo:<repo>:ref:<ref>” for branches. This can lead to some wonky config files, particularly if you want to authorize pull requests differently than protected branches. GitHub even provides organizational policies for custom subject templates which, while undoubtedly useful, seems especially cursed.
For Oblique, validating the token is easy using go-oidc. Here’s ~20 of lines of code to do that:
// Public keys for verifying signatures and other metadata are queried
// using this issuer URL.
const actionIssuer = "https://token.actions.githubusercontent.com"
provider, err := oidc.NewProvider(ctx, actionIssuer)
if err != nil {
	// ...
}
config := &oidc.Config{
	// MUST match the "audience" used when minting the token.
	ClientID: "oblique",
}
verifier := provider.Verifier(config)

// Verify the signature, issuer, audience, and expiry of the token.
idToken, err := verifier.Verify(ctx, rawIDToken)
if err != nil {
	// ...
}

// For a complete list, see:
// https://docs.github.com/en/actions/reference/security/oidc#custom-claims-provided-by-github
var claims struct {
	// GitHub username that triggered the action.
	Actor string `json:"actor"`
	// The name of the branch in the form "refs/heads/".
	Ref string `json:"ref"`
	// The full name of the repository, of the form ":".
	Repository string `json:"repository"`
	// Will be set if using deployment environments.
	Environment string `json:"environment"`
}
if err := idToken.Claims(&claims); err != nil {
	// ...
}
// Use claims to determine if the Action is authorized.
Based on the branch and repo fields, we can then make decisions to allow or deny access to our API without any copy and pasting of API keys.
Workload identity
It’s rare that a good security outcome provides a significantly better experience for users. Workload identity is absolutely one of those exceptions. You want users to be able to say “please let this action use Terraform to manage the service” without juggling API keys or figuring out how to store those credentials.
While I can (and will) gripe about some of the specifics about the implementation, we shouldn’t just ask for but expect identity primitives like this from any CI/CD, cloud, or infra product. If a platform lets you run code, it should be able to authenticate itself to an external system without pre-configuring an API key.


Good justifications write themselves
Maya Kaczorowski — Fri, 25 Jul 2025 15:00:00 GMT
Your authorization system needs context about why users have access to resources. You might think this is just for compliance — user access reviews, audit trails, the usual checkbox exercises. But context matters more than that.
Access should be explainable. When you understand why someone has access, you can make confident decisions about managing it. It’s hard to know whether it’s safe to remove access if you don’t even know why it’s there to begin with. When you’re reviewing audit logs, meaningful context helps you quickly understand what changed and why.
The typical implementation is a “justification” field on access requests. A user wants Salesforce access via Slack, they get a text box. What do they actually write?
Please give me access to Salesforce
working on project Alpha
I’m a new SDR
asdf
This isn’t the context you’re looking for. This is users trying to get past your form validation so that they can get back to work. And this information decays: we canned Project Alpha two years ago (RIP) and Leo has been promoted to Account Executive. (Congrats, Leo!)
Some organizations try structured justifications with regex patterns:
bug #12345
Acme Inc. support ticket #ABC123
This works well when access is tied to specific work items. If the only reason an individual needs access is due to a specific bug or support ticket, this works great! But who says your ticket has sufficient context and isn’t just yet-another-text-box? Most access requests aren’t about one-off tasks — they’re for routine work that happens every day without a corresponding ticket number.
Most justification should be inferred. If someone works in Sales, of course they need Salesforce access. If they’re on the infrastructure team, they probably need AWS access. You already have this context — job titles, team membership, project assignments. Use it.
You’ll still need that freeform text field as an escape hatch for truly exceptional cases. But recognize it for what it is — a last resort that will generate lower-quality context than the inferred or structured justifications you can implement for most scenarios.
Good context doesn’t have to come from a literal justification field. When you’re doing access reviews or investigating changes, you should already have enough information to understand why access exists. If you don’t, fixing your role definitions and team mappings will serve you better than asking users to fill in more text boxes.
Context is essential for explainable access, but you have multiple ways to get it. Infer it from existing organizational data whenever possible. Use structured justifications for specific work items. Reserve freeform explanations for genuine exceptions. Design your permissions around these different sources of context, and you’ll spend less time chasing users for explanations that don’t actually explain anything.


Chesterton's fence doesn't apply to access controls
Maya Kaczorowski — Fri, 27 Jun 2025 15:00:00 GMT
IT teams reluctantly apply this principle to access controls. They’re terrified of removing access because they don’t understand why it was granted in the first place. What if removing it breaks something? This fear consistently trumps the principle of least privilege, leaving organizations with sprawling access that nobody understands. Why not apply Occam’s razor instead? The simplest explanation for why someone has admin access to everything is that they must need it, right?
But access controls work differently than fences.
Use the data you already have
The fear is real: remove the wrong access and someone can’t do their job (or the cows escape). But just because you don’t understand why access was granted in the first place (the core issue in Chesterton’s fence) doesn’t mean you can’t validate that it’s safe to remove.
Unlike rebuilding a fence, regranting access should be straightforward — except that in most organizations, granting access can take days of approvals that everyone dreads. If we can make getting back that access really really easy, then is that such a big deal? The fear diminishes when the recovery process is painless.
And unlike mysterious fences, access controls generally have logs. Use them. You might not know why someone got access to customer data many years ago — and the person who granted that access or built that fence might be long gone — but you can see the last time that access was used. Preserving unused access isn’t security, it’s digital hoarding.
Removing access that hasn’t been used in the last 90 days likely has low risk of actual business impact. This is where most access cleanup should start. Don’t try to understand the original intent behind every permission: look at what’s actually being used and start there.
Context still matters
The core insight of Chesterton’s fence — that context matters — absolutely applies to access controls. Good access management isn’t about preserving historical decisions — it’s about making informed decisions based on current data. You can’t make your access controls more manageable, and move towards the principle of least privilege, without context.
The best context to have is why access was originally granted. Without capturing that context, every change feels risky. But we often have the next best thing: current usage patterns help us understand the actual risk of removal. Sometimes the fence really was just someone’s mistake, sitting there for years because everyone was too afraid to remove it. (Why do I have access to Salesforce? We’ll never know…)
Stop preserving access out of fear. Start with what’s unused, make recovery fast, and gradually build from there. The biggest risk isn’t removing the wrong access — it’s never removing any access at all.


Identity management is harder than it should be
Maya Kaczorowski — Mon, 23 Jun 2025 15:00:00 GMT
Our thinking on identity hasn’t evolved in a long time. The predominant access control model of Role-Based Access Controls (RBAC) was developed in the 1990s, while the more modern alternative of Attribute-Based Access Control (ABAC) is already a decade old. We’re having the same conversations now we’ve been having for years: Should access be granted on demand? How should you manage service account identities? What happens when someone’s manager changes? Most organizations still struggle to implement sane access controls.
The problem isn’t missing frameworks or standards. We have plenty of those. The problem is that identity management in practice looks nothing like identity management in theory.
Identity should be simple
Most organizations aim for a reasonable approach that should, in theory, ensure only authenticated and authorized users can access resources: authenticate using SSO through an identity provider, ideally with hardware-backed MFA, and sync users and groups with SCIM provisioning. When fully implemented, these controls work well. The challenge is that any gaps in coverage — whether it’s legacy applications that don’t support SSO or SCIM, incomplete SAML logouts, or weak MFA — create openings that attackers consistently exploit. Groups like LAPSUS$ have taken advantage of these gaps again and again. Which is to say, these controls don’t work in reality.
We don’t live in a world where every application supports these standards. Critical apps like Stripe only support SSO in beta and don’t support SCIM at all. Snowflake didn’t require MFA until recently. Forget social media accounts — getting to 100% coverage is nearly impossible when even security-conscious SaaS providers aren’t there yet.
SaaS has grown faster than we can secure it and very much became the default. Your network and endpoints still matter, but identity has become the primary security perimeter — the only control point that touches every application your employees use.
Identity should be simpler
You can’t simply apply SSO, SCIM, and MFA everywhere because not every vendor supports these standards equally, or at all. We live in a world of bandaids — that’s why we have password managers.
Fragmentation makes coverage gaps worse: we end up with multiple ways to solve the same problem, none of them particularly well established. Should authorization be enforced through SSO or SCIM provisioning? There’s no central place to manage authentication and authorization, and since they’re tightly coupled, you can’t treat them as separate problems — yet every vendor implements them differently. This forces every organization to build more identity and security tools, such as proxies to access GitHub or jump hosts to authenticate users via SSH, because the underlying identity systems can’t work consistently across all applications. You need tools to discover which services employees signed up for, and more tools to audit which permissions they have in those services.
We’re not trying to discourage you from implementing reasonable identity controls: you can, and should, get critical production apps behind SSO and MFA, using a proxy if needed to meet those requirements. This is still the right goal — it’s valuable, necessary work that belongs on your IAM roadmap. But you also can’t escape the treadmill of slowly forcing every new SaaS application you buy to meet the same requirements. Even organizations with internal requirements to only purchase SaaS apps at the tier that includes SSO (damn that SSO tax!) can’t get full compliance.
Access controls are constantly changing, because organizations are constantly changing
Identity is hard to get “right” because it’s a moving target that never stops moving.
Not only do access controls need to change often, they need to change quickly. Sure, you should remove unused access (but who does?) and revoke access when employees leave, but it’s much more important to ensure employees have access when they need it. No one wants multiple back-and-forths with IT to get the access they need to do their job. This wouldn’t be so difficult except that this change is constant: there’s always another ticket, another role change, another exception.
Identity is gardening, not engineering. There’s constant weeding and replanting. It’s highly manual work that doesn’t scale the way other security domains do.
Making it scale requires acknowledging what makes it unique. You can’t write generic policies that work for every organization because every organization is unique. Not just different tech stacks, but different organizational structures. A startup’s flat hierarchy needs different controls than a regulated enterprise with strict separation of duties. People and culture are what make an organization what it is, and this business context must influence how we think about security.
Access decisions require context that security teams don’t have
Most organizations want to follow the principle of least privilege: that employees should only have access to what they need. But how do you determine that? We don’t know what someone should have access to. IT and security teams lack the business context for these decisions: What is this app? Is it sensitive? What kinds of users should have access to it? This context exists primarily in people’s heads, and if documented, it’s in an unmaintained spreadsheet. Context decays quickly.
When you don’t know what access users need, you make a mess. Users over-request access to unblock themselves. IT copies someone else’s permissions hoping it works. These incremental changes accumulate into inconsistent controls that never get cleaned up. You end up with hundreds of unused groups and permissions that made sense three reorganizations ago — and no one knows what’s actually needed anymore.
When you get provisioning wrong initially, you spend time cleaning it up later. Debugging why someone’s access works or doesn’t is nearly impossible. You fear removing access and breaking the business — especially when granting access again requires another ticket and hours of waiting. The industry has invented whole product categories to address this, like IGA for user access reviews required by compliance and ISPM for identifying IAM misconfigurations. But these are symptoms of the underlying problem: that we don’t have a systematic way to maintain access controls as organizations evolve.
Building something better
There’s no shortage of identity pain points. Even basic controls like SSO and MFA are nearly impossible to implement everywhere. Access requirements constantly change, making them difficult to maintain. And identity decisions require business context that security teams rarely have.
These aren’t just theoretical problems. They have real, measurable impact on people’s daily work. IT teams spend countless hours on access request tickets that could be automated. Employees get frustrated when they can’t access the tools they need to do their jobs. Security teams burn out from the constant manual work of managing exceptions and reviewing access that may or may not be appropriate.
When we founded Oblique, we set out to work on real, tactical, impactful security problems that aren’t disappearing anytime soon — and that’s what we’re doing. Identity is a mess right now, but identity was also a mess before, and it’ll probably keep being a mess.
We’re tackling authorization for corporate environments first. It’s not trendy, but we know it’s impactful, because it’s what we heard security leaders complain about most. Your IT and security teams want their access controls to be more maintainable. We’re here to help you get there.