Deploy AIDR Container on AWS

These installation steps show you how to install locally into a Minikube cluster. Instructions are also valid for Kubernetes provided by Docker Desktop and a full-fledged Kubernetes cluster. Running locally with Minikube introduces extra latency due to the prompt injection classification model. Latency is not present in a full-fledged Kubernetes cluster.

Before deploying Model Scanner, make sure you understand:

• Resource Requirements
• Hybrid and Disconnected modes
• Proxy and Prompt Analyzer

Select OS

Select your operating system and deployment type to view installation instructions.

macOSWindowsUbuntu

HybridDisconnected

A Hybrid deployment sends metadata to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   cat << EOF > env
   HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   EOF
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Load the environment variables into your shell.

   source ./env
   

3. Create a file named .env.local.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <client id> and <client secret> with your HiddenLayer API key and secret.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   cat << EOF > .env.local
   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     HL_LLM_PROXY_CLIENT_ID=<client_id>
     HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     # HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   EOF
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop is running.

5. Run the following command to login to the HiddenLayer Quay repository.

   docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

A Disconnected deployment doesn't send data to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   cat << EOF > env
   HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   EOF
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Load the environment variables into your shell.

   source ./env
   

3. Create a file named .env.local.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   cat << EOF > .env.local
   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     # HL_LLM_PROXY_CLIENT_ID=<client_id>
     # HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   EOF
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop is running.

5. Run the following command to login to the HiddenLayer Quay repository.

   docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

HybridDisconnected

A Hybrid deployment sends metadata to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Create an environment variable file with Quay credentials. Manually create an env.bat file with the following content. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   @echo off
   set HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   set HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Open a command prompt and load the environment variables into your shell.

   env
   

3. Create a file named .env.local, then copy and paste the following example into the file.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <client id> and <client secret> with your HiddenLayer API key and secret.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     HL_LLM_PROXY_CLIENT_ID=<client_id>
     HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     # HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop and the Kubernetes cluster are running.

5. Run the following commands to login to the HiddenLayer Quay repository.

   docker login --username=%HIDDENLAYER_USERNAME% --password=%HIDDENLAYER_PASSWORD% quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

A Disconnected deployment doesn't send data to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Create an environment variable file with Quay credentials. Manually create an env.bat file with the following content. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   @echo off
   set HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   set HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Open a command prompt and load the environment variables into your shell.

   env
   

3. Create a file named .env.local, then copy and paste the following example into the file.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     # HL_LLM_PROXY_CLIENT_ID=<client_id>
     # HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop and the Kubernetes cluster are running.

5. Run the following commands to login to the HiddenLayer Quay repository.

   docker login --username=%HIDDENLAYER_USERNAME% --password=%HIDDENLAYER_PASSWORD% quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

HybridDisconnected

A Hybrid deployment sends metadata to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   cat << EOF > env
   HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   EOF
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Load the environment variables into your shell.

   source ./env
   

3. Create a file named .env.local.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <client id> and <client secret> with your HiddenLayer API key and secret.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   cat << EOF > .env.local
   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     HL_LLM_PROXY_CLIENT_ID=<client_id>
     HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     # HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   EOF
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop and the Kubernetes cluster are running.

5. Run the following command to login to the HiddenLayer Quay repository.

   docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

A Disconnected deployment doesn't send data to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

1. Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

   • For Quay credentials and other requirements, see Resource Requirements.

   cat << EOF > env
   HIDDENLAYER_USERNAME=%QUAY_USERNAME%
   HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
   EOF
   

   Special Characters

   Special characters will require wrapping the value in a string.

2. Load the environment variables into your shell.

   source ./env
   

3. Create a file named .env.local.

   • Use the comments in the example to find configurations for your deployment needs.

   • File changes to note:

     • For the EU region, uncomment hl_region=eu.
     • Replace <license> with your AIDR license.
     • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

   cat << EOF > .env.local
   namespace:
     name=aidr-genai
   
   image:
     ## By default, the image below is deployed; this image is for CPU deployments
     repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
     ## To set up a GPU deployment, uncomment the line below to use the CUDA image
     # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
     ## By default, the latest version is installed
     ## Uncomment the following to specify a specific version
     # tag=25.5.2
   
   ## By default, hybrid connection will connect to the 'US' region
   ## Uncomment the following option to change to a different supported region  
   # hl_region=eu
   
   resources:
     ## For CPU deployments, 8 vCPUs are recommended
     ## For GPU deployments, 4 vCPUs are recommended
     requests:
       cpu=8
     ## For GPU deployments, uncomment this to allocate 1 GPU per instance
     # limits:
       # nvidia.com/gpu=1
   
   ## Set number of instances for deployment
   replicas:
     min=1
     max=1
   
   ## Specify a specific node group if necessary
   ## For instance, GPU deployments may require specific nodes to access a GPU
   # nodeGroup=<nodegroup_name>
   
   config:
     HL_LICENSE=<license>
     HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
     HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
   
     ## This must match the number of vCPUs assigned above
     OMP_NUM_THREADS=8
   
   ## By default, the hybrid connection type is used and requires a client id and secret
     # HL_LLM_PROXY_CLIENT_ID=<client_id>
     # HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
   
     ## Uncomment the following to disable the connection to the AISec Platform
     ## in this mode, the client id and secret are not required
     HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
   EOF
   

   Policy Configuration Environment Variables

   For other policy configuration environment variables, see AIDR Configuration.

4. Make sure Docker Desktop and the Kubernetes cluster are running.

5. Run the following command to login to the HiddenLayer Quay repository.

   docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io
   

6. Run the AIDR container.

   docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest
   

Health Check

To check that the proxy is up and running, open a new terminal and use the following command to ping the health monitor.

curl http://localhost:8000/health

Generate Manifest

To write the generated manifest to a file (manifest.yml), open a terminal and run the following command.

• You must run the AI Detection & Response installer before generating a manifest.
• Change latest to the AI Detection & Response version that you use.

docker run --env-file ./env -v pwd/config:/src/config quay.io/hiddenlayer/distro-enterprise-aidr-genai-installer:latest > manifest.yml