Hash query language

 ___  ___  ________  ________  ___  ___          ________  ___  ___  _______   ________      ___    ___      ___       ________  ________   ________  ___  ___  ________  ________  _______      
|\  \|\  \|\   __  \|\   ____\|\  \|\  \        |\   __  \|\  \|\  \|\  ___ \ |\   __  \    |\  \  /  /|    |\  \     |\   __  \|\   ___  \|\   ____\|\  \|\  \|\   __  \|\   ____\|\  ___ \     
\ \  \\\  \ \  \|\  \ \  \___|\ \  \\\  \       \ \  \|\  \ \  \\\  \ \   __/|\ \  \|\  \   \ \  \/  / /    \ \  \    \ \  \|\  \ \  \\ \  \ \  \___|\ \  \\\  \ \  \|\  \ \  \___|\ \   __/|    
 \ \   __  \ \   __  \ \_____  \ \   __  \       \ \  \\\  \ \  \\\  \ \  \_|/_\ \   _  _\   \ \    / /      \ \  \    \ \   __  \ \  \\ \  \ \  \  __\ \  \\\  \ \   __  \ \  \  __\ \  \_|/__  
  \ \  \ \  \ \  \ \  \|____|\  \ \  \ \  \       \ \  \\\  \ \  \\\  \ \  \_|\ \ \  \\  \|   \/  /  /        \ \  \____\ \  \ \  \ \  \\ \  \ \  \|\  \ \  \\\  \ \  \ \  \ \  \|\  \ \  \_|\ \ 
   \ \__\ \__\ \__\ \__\____\_\  \ \__\ \__\       \ \_____  \ \_______\ \_______\ \__\\ _\ __/  / /           \ \_______\ \__\ \__\ \__\\ \__\ \_______\ \_______\ \__\ \__\ \_______\ \_______\
    \|__|\|__|\|__|\|__|\_________\|__|\|__|        \|___| \__\|_______|\|_______|\|__|\|__|\___/ /             \|_______|\|__|\|__|\|__| \|__|\|_______|\|_______|\|__|\|__|\|_______|\|_______|
                       \|_________|                       \|__|                            \|___|/

# ABOUT

Based on the grammar for Kusto Query Language, Hql is designed to break through the barriers restricting detection engineering and threat hunting (DEATH). It acts as a hybrid compiler and interpreter, enabling a full featured and extensible query language to effectively operate on, and across, any backend database. This allows for seamless detection engineering without having to worry about managing detections, converted or bespoke, yourself.

Hql iterates on existing ideas like Sigma to provide a completely generic representation of a detection query complete with documentation. Hql differs however in that the detection documentation is carried at all times with the code. That is, where a converted Sigma query would have its documentation separated from the actual query itself upon conversion, Hql maintains the documentation as a deoxygen-style comment with it at all times.

Provided as a python library and engine, Hql can serve as a one off terminal ran command, or as a multi-threaded daemon which schedules detections directly. Designed to be simple to run, such as needing to quickly set up a detection engine on a laptop for an engagement, there's no need for an over engineered deployment. Hql is designed to be ran as a single python module using Python 3.14t to provide multi-threading. This is provided in a prebuilt single container if you lack a free threaded version of Python, or if needed, as low as Python 3.9 without multi-threading.

# MOTIVATION

I came up with the idea back on March 7th 2025. I was frustrated that I, a homelabber, and a DFIR/DEATH person was not able to hunt on my own systems. I was and still am using Graylog (as a DEATHCON 2024 workshop) and found that it was impossible to actually get anything done in it. Around this time I was looking for a good self project I could do to improve myself, so I wrote down ideas, and stopped at the first one:

Take in a kusto query and process it using elastic search.

The only other projects (that I know of) that were Trino/PrestoDB, Dremio, and (ironically) Hibernate Query Language. I'm not paying for propriety software, and Trino is Java. Hibernate, also java, also only operates on SQL databases with a JDBC connector, and a exiting dialect. SQL can be difficult to write, and SIEM data isn't always SQL, as in Graylog, making it a no go. Also they're way too complicated and don't really do what I want, or in a language I want. So the idea was to bridge the gap between free cybersecurity tooling, and expensive tooling such as Sentinel or Splunk.

I'm used to Kusto, and I think it's an extremely expressive language, hence my choice. I also took inspiration in my C programming history and decided to use deoxygen-style comments once I realized this could evolve into DaC, which I've called HaC to distinguish. It needed to be dead simple to run and effectively have no restrictions, and I think I've achieved that or at least provided the ground work.

The result is a database/SIEM agnostic Kusto dialect called Hash query language. Much much more needs to still be done, but I hope you enjoy my humble PoC.

# FEATURES

Backend database agnostic compilation
Implements a uniform feature set across all databases
Designed with detection as code in mind
Easy to run, requiring only 3.9
Multi-threaded detection engine with Python 3.14t
Direct Sigma support
Native compiler-level field mapping support
Easy ability to query data, tranform, and push to another DB/Format
Not AI driven!!!!!!!
No AI code!!! (except for the PoC interface, engine is completely organic)

# QUICK START

# podman or docker if you don't have 3.13t or 3.14t
$ podman pull hashfastr/hql
$ podman run --rm -v "$PWD:/data:z" hashfastr/hql --init

# Hello world
$ podman run --rm -v "$PWD:/data:z" hashfastr/hql -v -f /data/examples/hello-world.hql

# Start the detection engine 
$ podman run --rm hashfastr/hql -eng -d ./detections

# Convert a sigma query
$ podman run --rm -v "$PWD:/data:z" hashfastr/hql -dpar -f /data/sigma-rule.yml

# EXAMPLE QUERIES

Direct sigma conversion

/**
 * @title Set Files as System Files Using Attrib.EXE
 * @id bb19e94c-59ae-4c15-8c12-c563d23fe52b
 * @status test
 * @schedule 0 * * * *
 * 
 * @description
 * Detects the execution of "attrib" with the "+s" flag to mark files as system files
 * 
 * @author frack113
 * @related
 * - {'id': 'efec536f-72e8-4656-8960-5e85d091345b', 'type': 'similar'}
 * 
 * @references
 * - https://github.com/redcanaryco/atomic-red-team/blob/f339e7da7d05f6057fdfcdd3742bfcf365fee2a9/atomics/T1564.001/T1564.001.md#atomic-test-3---create-windows-system-file-with-attrib
 * - https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/attrib
 * - https://unit42.paloaltonetworks.com/unit42-sure-ill-take-new-combojack-malware-alters-clipboards-steal-cryptocurrency/
 * 
 * @date 2022-02-04
 * @modified 2023-03-14
 * @tags
 * - attack.defense-evasion
 * - attack.t1564.001
 * - detection.threat-hunting
 * 
 * @falsepositives
 * - Unknown
 * 
 * @level low
 */
product('windows').category('process_creation')
| where Image endswith '\\attrib.exe' or OriginalFileName in ('ATTRIB.EXE')
| where CommandLine contains_any (' +s ')

Direct database access and improvements

let Processes = database('elastic').index('so-beats*')
| where event.code == 1
| where ['@timestamp'] > ago(1h)
| extend Filename = process.pe.original_file_name, Hostname = host.name
| summarize count(), makemv(Hostname) by Filename
;
let RDS = database('rds-sqlite').macro('process-info')
| extend Filename = filename
;
Processes
| join RDS on Filename
| where malicious == True
| project Hostname, Filename