security
What we hold, and how
GPU Zero runs your training in your cloud and keeps its records, so it holds keys to your cloud, your code and your runs. Here is how each is guarded, and what is not done yet.
Your cloud and your bill
- GPUs are rented only on the provider account your workspace connects, and billed there. GPU Zero has no provider account of its own and resells nothing.
- Your provider keys are stored encrypted, never shown back, never sent to a machine, and used only to make and remove that workspace's own nodes. A key is tried before it is kept.
- A monthly budget is applied by our server while it is up: at the budget, running jobs stop with a final checkpoint and their machines are given back, and no new one is made. Per-node and per-hour limits apply on top. The spend it counts is an estimate from the machines it made, not your provider's bill, so set a limit at your provider too.
Signing in
- Sign-in is handled by Auth0. GPU Zero never sees or stores a password.
- Every request is checked twice: by the gateway in front of the API, and again by the API itself, which trusts nothing the gateway adds.
- The actions a stolen session could turn into lasting access - making a machine token, changing a provider's keys or the storage bucket, deleting the account - ask for a sign-in from the last ten minutes.
- Signing out ends every session of the account, on every device.
Tokens and secrets
- A machine token is stored only as a hash and shown once. Settings lists each one with when it was last used, and revokes it at once.
- The secrets you give your runs are encrypted at rest, never shown again, and handed only to the jobs launched with them. Launch selects all of a workspace's secrets for a job unless you untick some, and on a marketplace such as vast.ai the machine belongs to a third party who can read what a job is given. A value a script prints is replaced by the secret's name before the log is stored.
- Everything belongs to a workspace, and every request is answered only from the caller's: another workspace's run, job or node answers as if it did not exist.
What runs next to your model
- The agent on a node runs a script your repository declares in gpuzero.toml, at the commit you chose, and never a command it was sent. Parameters become flags whose names and values are checked.
- A private repository is read through the GPU Zero GitHub App with a read-only token for that one repository, which expires within the hour, or with a token you keep as a secret. The training process never receives it.
- The agent, Ravex and Moonclip are open source under Apache-2.0: what runs on your machines can be read line by line.
Your data
- Checkpoints are written by the training process itself, to the S3 or R2 bucket you connect, or, without one, to a bucket GPU Zero operates for your workspace. The servers that run the console never hold model weights.
- Every run can be downloaded whole, as one file, at any time.
- Who else handles your data, and how long it is kept, is in the privacy policy.
Not there yet
GPU Zero is young, and some of what a company expects does not exist yet:
- Single sign-on (SAML or OIDC for your company) and directory sync.
- A second factor required for every account, and an audit log of who did what in a workspace.
- Roles inside a workspace: today every member can do everything.
- A self-hosted or private deployment of the platform.
- A SOC 2 report or any other certification, a service-level agreement, and a public status page.
Reporting a problem
If you find a security problem, write to security@gpuzero.dev before saying it anywhere else. You will get an answer, and credit if you want it.