Onyx Lands on palmer-ai, and Railway Tells Us Three Things It Doesn't Do

Why Care?

Onyx has sat on the watchlist since July as "31k stars, verify the license." It is now deployed for palmer-ai: nine Railway services, an object-storage bucket, and a public URL that answers. Its license turned out to be MIT Expat with ee/ directories carved out — GitHub's "Other/NOASSERTION" was never the real answer, and the watchlist row now says so.

The more durable output is not the deployment. It is three Railway constraints that were not written down anywhere in this repo, each of which will bite the next bundle that isn't a plain web app.

Railway does not apply start-command arguments to image-based services

This cost more time than the other eight services combined, and it failed twice wearing different masks.

alembic upgrade head && uvicorn onyx.main:app … ran the alembic half, wrote two hundred migrations, and then stopped. The container did not crash. It sat at the image's default CMD ["tail","-f","/dev/null"], burning 6MB of RAM and reporting SUCCESS in the dashboard. A green deployment that is running a no-op keepalive is a genuinely nasty failure mode.

Then it reappeared as a redis problem. redis-server --requirepass <secret> produced a server where CONFIG GET requirepass returns empty. Onyx dutifully sent AUTH, redis said "no password is configured for the default user," and onyx-api died — but the visible error was ten repetitions of Document index OpenSearchIndexPair setup did not succeed, because that setup takes a redis lock before it touches OpenSearch. Chasing the OpenSearch message would have been chasing a symptom two layers from the cause.

The rule now recorded: never rely on &&, $VAR, or extra CLI flags in a Railway start command for an image-based service. Migrations go in pre_deploy_command. Config reaches the process through env vars or a config file baked into a wrapper image. And one command settles it in five seconds:

railway ssh --service onyx-redis "redis-cli config get requirepass"

We took the upstream-matching fix — Onyx's own compose runs redis without auth — rather than gold-plating a wrapper image mid-deploy. Hardening is logged as a follow-up, not pretended away.

Railway's private network is IPv6; Onyx binds IPv4

*.railway.internal resolves to IPv6, so a process bound to 0.0.0.0 is simply unreachable. The tell is that it hangs rather than refuses — curl returns 000, not a connection error, because nothing is there to send a RST. Once the bind is right but the process is down you get a fast 502. That 000-versus-502 distinction turned out to be the single most useful diagnostic of the whole deploy, and it is now in the runbook.

The model servers were the interesting case: model_server/main.py hardcodes host = "0.0.0.0" with an explicit comment and no env override. But app is a module-level object, so driving uvicorn directly against model_server.main:app --host :: sidesteps the hardcoding without patching upstream.

Railway volumes mount root-owned; OpenSearch refuses to run as root

The stock image fails with AccessDeniedException on its data directory. The obvious fix, RAILWAY_RUN_UID=0, fails with OpenSearch cannot run as root. Neither knob can work alone — worth writing down precisely because the first one is what everybody tries.

The fix is a thin wrapper that starts as root, chowns the mount, drops to UID 1000, and execs the upstream entrypoint unchanged. It lives in committed docs/onyx/railway/opensearch/, not in gitignored client-stacks/. There was a precedent for the latter — the-water-foundation has a Dockerfile.railway sitting in an untracked directory — and that precedent is a trap: an image you cannot rebuild from the repo is not infrastructure, it is a souvenir. Any client deploying Onyx needs this wrapper, so it belongs where every client can reach it.

Two details that ate cycles anyway: setpriv is in the full util-linux package, not util-linux-core; and discovery.type=single-node is load-bearing because it skips the bootstrap checks Railway cannot satisfy (vm.max_map_count is a host sysctl, not a container setting).

nginx had to be deployed, not dropped

Onyx serves one origin split by path — /api, /openapi.json, /scim and /auth/saml to the backend, everything else to Next.js. Railway's edge maps a domain to a single service and cannot path-route, so upstream's nginx becomes a real Railway service and the only one with a public domain.

Our version makes one improvement over upstream's config. Upstream uses upstream {} blocks, which nginx resolves exactly once at startup — so a cold start crashes with "host not found in upstream," and an onyx-api redeploy strands the router on a dead IP. Ours puts the target in a variable and points resolver at Railway's own nameserver, lifted from /etc/resolv.conf at boot. Resolution happens per request. It came up cleanly with both upstreams still nonexistent, which is exactly the behaviour we wanted.

What this costs, said plainly

~$100–200/month for palmer-ai alone — an order of magnitude above Twenty and Outline combined. OpenSearch holds a 2GB JVM heap permanently; both model servers keep embedding models resident.

Railway is the right host for a one-to-two month trial: it matches the stack's conventions, the ops tooling, and the backup story, and nothing about it is one-way. It is not the cheap long-term home. A fixed-price box — Hetzner CCX23, 4 dedicated vCPU and 16GB, about €25/month — runs upstream's compose unmodified for roughly a fifth, and is the path Onyx actually documents and tests. Onyx's state is Postgres plus OpenSearch plus object storage, all portable. The runbook exists so that migration stays cheap if the trial earns it.

Not done yet

The instance is live but not claimed: AUTH_TYPE=basic with has_users:false means the first signup becomes admin, and it is sitting on a public URL. That needs closing today. No LLM provider is configured, so the chat surface is inert until one is added. Connector ingest and the S3 upload round-trip are unexercised — the Railway bucket is virtual-host style, which boto3 can be particular about — and the code-interpreter service was not deployed at all.

Verified working end to end: /nginx-health, /api/health returning {"success":true}, the web UI at 200, OpenSearch GREEN, and Celery beat scheduling on its own.