Protection Against Internal Threats(e.g., rogue admins or engineers)

High: Data remains encrypted at the field/column level unless accessed via an authorized identity through Ubiq. Even Databricks workspace admins or cloud engineers cannot see plaintext sensitive data without going through Ubiq’s IAM-governed decryption process. Encryption keys and logic reside outside of Databricks, so a malicious insider cannot retrieve secret data by querying Delta tables or storage alone.

Result: If a DBA or platform admin tries to read a protected Delta Lake table directly (or even access the raw files on S3/ADLS), they get only ciphertext; Ubiq’s external service must approve and provide keys for any decryption. This effectively thwarts “inside jobs” because encryption isn’t lifted automatically inside the cluster – it stays tied to user identity and policy at all times.

Low: Relies on in-platform access controls and trust of privileged users. Databricks provides identity-based permissions (Unity Catalog table ACLs, row-level filters, etc.), but once a user is authorized, data is delivered in plaintext. A determined insider with high privileges (workspace admin, cluster owner, or cloud storage admin) can typically find a way to access data. For example, a metastore admin can grant themselves access or an admin with cloud credentials could read raw data files from storage if misconfigurations exist. Out-of-the-box encryption at rest is meant for physical data theft, not limiting insiders – if the system is up and running, any allowed query returns decrypted data. Unity Catalog and workspace ACLs deter casual snooping, but do not inherently prevent data from being decrypted inside a running cluster by those with sufficient rights. There are audit logs and some admin controls, but fundamentally a privileged insider (or malware running with their rights) can access clear data by design. In short, Databricks’ native model assumes trusted insiders; it lacks a hard cryptographic barrier against a malicious insider with legitimate access.

Protection Against Credential Compromise(e.g., leaked tokens or stolen service principals)

High: Stolen Databricks credentials alone are insufficient to breach sensitive data protected by Ubiq. An attacker who gains a user’s Databricks personal access token or even compromises a Databricks cluster still cannot decrypt Ubiq-protected fields without also obtaining valid Ubiq authentication and keys. Simply having network or API access to the data plane yields only encrypted gibberish for protected columns. Ubiq enforces data access at runtime via an external IAM policy check, so a hacker impersonating a Databricks user or service account cannot surreptitiously decrypt sensitive fields unless they are also authorized in Ubiq’s system.

Result: A breach of Databricks credentials or a read-only cluster compromise does not equate to a breach of plaintext data – the attacker faces an additional identity barrier. This contains damage from token theft or leaked cloud keys, as the true decryption capability lies outside the Databricks environment.

Low: If an attacker compromises valid Databricks credentials (a user’s token, an admin’s login, or a service principal secret), they can generally access any data that identity is permitted to see. There is no secondary encryption layer tied to identity – data in Delta tables is stored in plaintext (assuming at-rest encryption which auto-decrypts for the service). Thus a stolen login or key gives the attacker whatever clear data that account could query. While good operational practice can limit each account’s scope, there’s no built-in cryptographic safeguard once an identity is authenticated. For example, if a service principal with access to critical tables is stolen, the attacker can use it to run notebooks or SQL queries and retrieve sensitive records in clear. Databricks does support features like short-lived tokens, user-specific cloud IAM roles (Credential Passthrough), and audit logs to mitigate misuse, but these don’t stop an attacker who actually holds a valid credential from reading data. In practice, stolen credentials = stolen data in the Databricks model, since the platform will dutifully return plaintext to any request with valid credentials. (By contrast, Ubiq’s model would require stealing a second set of credentials to get plaintext.)

Granularity of Access Control

Fine-Grained: Ubiq enables encryption and tokenization at a very granular level – down to individual fields, values, or data categories – with policies governing who (which user/role) can decrypt each piece of data. This means different users can have different views of the data: e.g. an analyst might only see masked or tokenized values, while a compliance officer with the proper role sees full plaintext. The granularity isn’t just content-based but identity-based: Ubiq uses the organization’s IAM roles and attributes to enforce field-level rules in real time. This ensures least privilege data access: even if two users query the same Delta table, one may get decrypted values and another gets ciphertext, depending on their entitlements. The control can be as fine as per-column per-user, or even conditional (based on context or attribute), far beyond traditional table-level permissions.

Fine-Grained (Policy-Based): Databricks’ Unity Catalog provides robust access control granularity in terms of permissions. Administrators can set table ACLs, define row-level filters, and apply column masking policies based on user roles or attributes. This allows segmenting data access so that, for example, a user only sees rows from their department and sensitive columns (like SSN) can be dynamically masked for unauthorized viewers. In effect, Databricks can deliver different subsets or masked versions of data to different users, achieving a form of fine-grained security at the access policy level. However, this granularity is logical rather than cryptographic – the data itself in storage isn’t partitioned by user or encrypted per role, it’s the enforcement layer in the query engine that applies these rules. The approach is powerful for governance, but its correctness relies on all access going through Databricks’ controls. If someone bypasses the engine (e.g., by reading files directly with sufficient cloud rights), or if an admin misconfigures a policy, the underlying data isn’t inherently segmented or encrypted by user. In summary, Databricks offers high granularity in permissions and masking, covering table, row, and column-level distinctions, but not granular encryption tied to identity.

Runtime Enforcement(Context-aware, policy-aware access in use)

Strong, Context-Aware: Every decryption request with Ubiq is checked at runtime against identity and policy (“Is this user/process allowed to decrypt this data now?”). The enforcement is in-line and external to Databricks – data only decrypts in the notebook or pipeline when the call to Ubiq’s service confirms the request meets the defined policy conditions. This means access decisions can incorporate context beyond just static roles – e.g. Ubiq could factor in the user’s current privileges, the specific dataset, or even other attributes like purpose or environment if configured. Critically, decryption is not automatic even after a Databricks query is authorized; the Ubiq client must actively request key release, which won’t happen if the policy says no. This provides an active guardrail during data use: even inside a running cluster, data stays encrypted until and unless the identity and context are validated. If a user’s permissions are revoked or context changes mid-stream, further decryption calls will fail – enforcing policies in real-time. Ubiq essentially wraps identity-based controls around data-in-use, not just at rest, making the security “live” and responsive to session context and identity state.

Limited to Identity-Based Rules: Databricks enforces access at query time based on the identity of the user executing the query and the policies in place. Unity Catalog’s engine will apply row filters and column masks dynamically for each query, ensuring that at runtime a given user only sees what they are allowed to see. This is a form of runtime policy enforcement, but it primarily considers who the user is (and what groups they belong to). Beyond identity and data attributes, there isn’t a built-in notion of richer context (such as time of day, query origin, or multi-factor authentication) influencing data access – the checks are mostly role/attribute-based and occur at query compile/execution time. Once the engine passes the query and the user has permission, the data returned to that user is plaintext and can be used freely within that session. The platform does not continuously interrogate context after that point. In other words, Databricks will enforce that only authorized users or groups can run certain queries or see certain rows/columns, but it won’t, for example, re-check policy on every single read operation beyond the initial query authorization. There is also an assumption that the execution environment is trusted: if a user is allowed, the data is decrypted in the cluster and could potentially be cached or passed to another function without additional checks. While effective for role-based access control, this lacks the deeper context awareness and external “always ask” key release that Ubiq employs. If someone manages to operate within the confines of an authorized session (or impersonate an authorized identity), Databricks will not apply further cryptographic breaks – the data is available to that session.

Integration Effort & Performance Overhead

Moderate Integration, Low Overhead: Using Ubiq in Databricks requires integrating Ubiq’s SDK or UDFs into notebooks and jobs – essentially instrumenting the data pipelines to encrypt and decrypt specific fields. This is an extra development step compared to using Databricks out-of-the-box, but it’s designed to be straightforward (Ubiq provides libraries for various languages). You typically need to identify sensitive fields in Delta tables and replace direct reads/writes with Ubiq encryption/decryption calls. This one-time setup per dataset yields strong security. Performance impact is minimal because Ubiq’s operations are lightweight and targeted only at sensitive data. Encryption/decryption occurs at the field level with efficient key retrieval, often incurring negligible latency. In practice, organizations find that Ubiq adds security without user-noticeable slowdown, since heavy analytics (scans/aggregations on non-sensitive data) are unaffected and only the confidential fields incur cryptographic processing. Ubiq’s design can even offload some work (e.g., key management) to its service, keeping the compute overhead on the Spark cluster low. The trade-off is the external dependency – the cluster must call out to Ubiq’s service for key access – but this is optimized and scalable. Overall, integration requires modest initial coding, but once in place, it operates seamlessly with minor performance overhead per protected field.

Built-in Features (Config Effort) & Some Overhead: Databricks’ native security requires configuration more than code changes. Enabling fine-grained controls might involve setting up Unity Catalog (with a metastore, catalogs, schemas), defining access policies (which can be a significant upfront policy-writing effort), and possibly configuring clusters to use Credential Passthrough or secure modes. There is an administrative overhead to manage permissions and policies over time, especially in large environments. The actual performance overhead of built-in security features is generally low: encryption at rest is handled by cloud storage or the platform with negligible impact on query speed, and in-transit encryption (TLS) is standard. Row-level filtering and column masking do add some query processing cost – each query has to evaluate filter UDFs or apply mask functions per row. In typical scenarios this overhead is modest, but complex masks or very large tables can see some slowdown due to these extra evaluations. Databricks’ query optimizer will prioritize security correctness over performance if there’s ever a conflict, which is the safe approach but might sacrifice some optimization opportunities. Overall, using native features is relatively frictionless for developers (no code changes to analytics logic; security is applied via platform), but it demands diligent admin effort to configure/maintain, and it introduces a slight runtime tax for dynamic policies. In exchange, you avoid external dependencies – all security is native to the lakehouse platform.

Data Type Coverage(Structured, Semi-structured, Files, Streams)

Broad Coverage: Ubiq’s data protection isn’t limited to tabular data – the same platform can protect structured data in tables, semi-structured data (JSON, logs) or even unstructured files, and real-time streaming data feeds. Within Databricks, Ubiq can integrate to protect Delta Lake tables (structured columns) as well as data flowing through notebooks or Spark jobs (e.g., encrypting a CSV being written out, or decrypting a JSON field on the fly). This means if your Databricks notebook reads sensitive data from a message stream or an object store file, Ubiq can be used to decrypt it just as with a Delta table. Likewise, if that notebook produces output files or sends data to an external system, Ubiq can encrypt those outputs, ensuring consistent protection beyond the lakehouse. In effect, Ubiq provides a uniform security layer across different data types and storage locations – one system to protect data in the lakehouse, in downstream files, or in Kafka streams, etc. This is especially useful if your sensitive data doesn’t live solely in Delta tables. The encryption travels with the data: even if someone exports a table to a CSV or moves data to another storage location, it remains encrypted/tokenized unless properly accessed via Ubiq.

Focused on Lakehouse Data: Databricks’ native controls cover data within the Databricks environment – primarily structured and semi-structured data stored in the lakehouse (Delta tables, Parquet/CSV files in data lakes) and data processed through its engine. For those, you have strong measures like table/column ACLs, and you can rely on cloud storage encryption for any file data at rest. Notebooks themselves (source code and results) can be encrypted at rest by Databricks and protected via workspace access controls. And streaming data can be secured in the sense that if you use Structured Streaming to write to/read from Unity Catalog tables, the same access policies apply in real-time. However, Databricks does not automatically protect data once it leaves the platform. If a user copies query results into a spreadsheet or moves data to an external system, the onus is on that user or system to protect it – the built-in encryption doesn’t persist beyond Databricks’ storage. There’s no native mechanism to encrypt data at the field level in exports or in-memory objects handed off outside. In summary, Databricks covers a wide range of data inside the lakehouse (any file format in DBFS or external object stores, structured or semi-structured, can be governed and encrypted at rest), but no coverage beyond its boundary. This is a key difference: Ubiq’s protections can travel with the data, whereas Databricks’ protections apply as long as data stays in-place on the platform.

Key Management Model

Externalized & Granular: Ubiq manages encryption keys in a separate, secure service – typically a multi-tenant cloud service or on-prem appliance under Ubiq’s control – that is FIPS 140-2 validated and designed for strong key security. Keys are never stored in plaintext on Databricks or in the data files; they are fetched just-in-time when an authorized identity requests decryption. Ubiq uses a hierarchical key model where each data set or even each column can have its own Data Encryption Key, which is then wrapped by a master key (which Ubiq secures internally, or optionally in the customer’s HSM/KMS if using BYOK). This per-field or per-dataset key approach means compromise of one key only affects that slice of data, and keys can be rotated regularly without re-encrypting everything or impacting applications (Ubiq handles rotation seamlessly). Key release is tied to identity policy: the Ubiq service will only release a key for decryption if the requesting user’s token is authorized for that data, adding an IAM-governed check every time. Even administrators of the Databricks environment cannot extract or misuse encryption keys, because they have no direct access to them – keys reside in Ubiq’s service. This external key management and wrapping model ensures that even if the Databricks cluster or storage is compromised, the data remains safe (attacker can’t get keys). It’s a strong, independent key security approach by design.

Platform-Managed Keys (Coarse or Hierarchical): Databricks primarily relies on cloud-provider key management for encrypting data at rest. By default, a Databricks workspace might use a single platform-managed key to encrypt all data at rest, but enterprises can opt for a customer-managed key (CMK) integration to have control. For instance, on AWS you can provide a KMS key that encrypts the workspace’s S3 storage and even the cluster VM disks. This gives you the option to revoke that key, rendering the data unreadable by Databricks – a sort of “big red button” for data if needed. Recently, Databricks introduced Multi-Key Encryption for Unity Catalog: a hierarchical key model that implements finer-grained encryption within the lakehouse. In this scheme, each Unity Catalog has its own encryption key, and each table (and even each file within a table) gets its own object key, which is encrypted by the catalog key. The root of this hierarchy can be your provided CMK or a Databricks key, and file-level keys are derived and not stored persistently. Benefit: If someone somehow obtains direct access to the stored files (bypassing Databricks), those file-level keys are not accessible, so the data remains gibberish. This improves isolation – compromise of one file doesn’t grant access to others. However, these keys are managed and used transparently by Databricks; they are not tied to individual user identities. When an authorized query runs, the platform automatically uses the necessary keys to decrypt data for that user. There’s no concept of “each user gets their own encryption key” – it’s per object, not per identity. Key management in Databricks is robust for protecting data at rest from external theft, especially with multi-key (it mitigates scenarios like an attacker directly reading cloud storage). But it’s not granular at the user/context level: the system unlocks data for any properly authenticated request, using its stored keys. Key rotation is supported (cloud KMS can rotate CMKs, and Databricks can re-encrypt with new data keys as needed), but doing field-by-field or frequent rotations might require re-writing data. In summary, Databricks’ model leverages strong encryption under the hood (and you can enforce that with your own master keys), yet the keys are ultimately controlled by the platform and applied uniformly to authorized access – not external IAM gating on each use.

IAM Compatibility (Enterprise Integration & Least Privilege)

Yes – IAM-Driven Data Access: Ubiq was built to integrate with enterprise Identity and Access Management. It ties into your existing IdP (e.g. Okta, Azure AD/Entra ID, LDAP) so that user identities and roles are consistently enforced. Access to decrypt data is governed by your central IAM policies: if a user’s role is revoked or their permissions change in the directory, Ubiq immediately reflects that (the user’s token would no longer allow decryption of those fields). This avoids a separate silo of accounts – Ubiq can use your Single Sign-On and group mappings, meaning least privilege principles from your IAM extend directly to data. Only those in the proper AD group, for example, could call Ubiq to decrypt a given field. Additionally, Ubiq’s model can support attribute-based rules (via IAM claims or group tags) for even more granular control. The result is a unified security stance: the same identities and roles that gate application access also gate data access at the cryptographic level. From an audit standpoint, this provides a clear, centralized way to manage who can see what data. Ubiq essentially acts as an enforcement arm of your existing IAM, rather than a parallel system.

Partial – IAM for Login, Roles for Data: Databricks integrates reasonably well with enterprise identity systems for authentication and user management. Customers often use SSO (SAML/OAuth) so that users log into Databricks with corporate credentials, and SCIM can sync enterprise user and group definitions into Databricks. Those synced groups can then be used in Unity Catalog policies, effectively linking corporate roles to data permissions. In that sense, Databricks leverages enterprise IAM groups to implement least privilege at the workspace and table level. Also, features like Credential Passthrough (on AWS and Azure) align Databricks with cloud IAM: when enabled, each user’s cloud identity and its IAM policies directly govern what data they can access in storage, rather than using a shared service account. This ensures, for example, that if a user isn’t allowed to access a particular S3 bucket by corporate policy, they can’t access it through Databricks either. However, there are limitations in IAM integration. The encryption of data is not IAM-contextual – once a user is authenticated, Databricks does not consult external IAM on a per-field decrypt (it just relies on the permissions configured in its own catalog). The coupling with IAM is at the coarse level of login and assigning groups, not at the fine-grained “decrypt this specific value only if Okta says this user has clearance.” Moreover, certain high-level roles in Databricks (like account admins) may bypass normal user-level restrictions for management purposes, which means in practice you must trust those roles (least privilege depends on how well you restrict admin assignments). Finally, while customer-managed keys in KMS give some IAM control (only certain cloud users can rotate or use the key), during normal operations the Databricks service assumes access to those keys to serve data. In summary, Databricks works with enterprise IAM for authentication and role mapping, but doesn’t natively leverage external IAM policies at the moment of data use beyond that. Achieving true least-privilege enforcement requires careful configuration of Unity Catalog permissions and cloud roles. There is no native concept of per-user encryption keys or decrypt approvals coming from an outside IdP in real-time – that is where a solution like Ubiq extends beyond Databricks’ built-in capabilities.