I have an amazon VPC owned by account A, shared with accounts B,C, and so on.
VPC CIDR 10.0.0.0/16
PrivateSubnet1 10.0.1.0/32
PrivateSubnet2 10.0.2.0/32
SomeSubnet 10.0.10.0/32
Route tables and all that should be just fine, several other processes are working on the network, like webservers and things in other subnets. Maybe not with RDS though?
PrivateSubnet1 and PrivateSubnet2 are in a subnet group. An RDS Postgres database is deployed in the shared VPC and specifically in the private subnet group.
Account B owns the RDS Postgres database, which is publicly available at the moment, but I'd like to eventually figure this out for private with SSH tunnelling from outside, and with apps on the EC2 instance.
Account C owns the EC2 instance, which runs a Jupyter notebook in the background (tmux) on an exposed port, as well as another detached service (webserver) through docker. I have a Caddy file which handles routing to the subdomains jup.mywebsite.com
and dockerapp.mywebsite.com
from an AWS Route 53 hosted zone mywebsite.com
.
I can SSH into my EC2 terminal, and I can runpsql -h my-db.xxxxxxxxxxx.us-west-2.rds.amazonaws.com -p 5432 -U postgres -W
where I am prompted for my password to enter the psql
command line. I can see databases/tables, create tables, and so on. So, my issue doesn't appear to be a firewall rule.
My Inbound security groups on both EC2 (Account C) and RDS (Account B) were opened up to All Traffic from network and specifically the exact IPs of the instance and RDS (assigned to 10.0.1.xxx according to nslookup
of the db_host.)
inbound security group for EC2
inbound security group for RDS
The blacked out IPs are just home and office IPs for myself and another developer.
So, if I enter Python from the EC2 command line
>>> import psycopg2>>> psycopg2.connect(database="my_db", user="postgres", password="********************", host="my-db.xxxxxxxxxxx.us-west-2.rds.amazonaws.com", port=5432)
it hangs then spits out Segmentation fault (core dumped)
and drops the Python session back to the EC2 command line.
Similarly, if I try to use SSHTunnelForwarder locally,
from sshtunnel import SSHTunnelForwarderimport psycopg2import os print ({os.getenv('EC2_SERVER_EIP')}, "\n", {os.getenv('SSH_USERNAME')}, "\n", {os.getenv('DB_HOST')}, "\n", {os.getenv('DB_NAME')}, "\n", {os.getenv('DB_USER')}, "\n", {os.getenv('DB_PASS')}, "\n", {os.getenv('DB_PORT')}) # all printing properly, so working belowwith SSHTunnelForwarder( (os.getenv('EC2_SERVER_EIP'), 22), ssh_username=os.getenv('SSH_USERNAME'), ssh_pkey="orchestration.pem", remote_bind_address=(os.getenv('DB_HOST'), 5432) #???) as tunnel: print("****SSH Tunnel Established****") conn = psycopg2.connect(database=os.getenv('DB_NAME'), host=os.getenv('DB_HOST'), #changing to '127.0.0.1' allows create table from local machine user=os.getenv('DB_USER'), password=os.getenv('DB_PASS'), port=tunnel.local_bind_port) #??? - hangs here!!! cursor= conn.cursor() table_creation = ''' CREATE TABLE table_from_local_with_ssh ( id SERIAL PRIMARY KEY, name TEXT NOT NULL )''' cursor.execute(table_creation) conn.commit() # cursor.close() conn.close()
the process hangs at print("****SSH Tunnel Established****")
in the psycopg2.connect process, with error
psycopg2.OperationalError: connection to server at "my-db.xxxxxxxxxxx.us-west-2.rds.amazonaws.com" (10.0.1.133), port 52530 failed: Operation timed out Is the server running on that host and accepting TCP/IP connections?
First off, where is that port (52530) coming from, is it ephemeral or something. I also tried without using tunnel.local_bind_port
, but hardset to "5432" didn't work.
Also, none of these approaches work from the jupyter notebook at jup.mywesbite.com, it just kills the jupyter kernel which I can then restart to try again.
I'm expecting the public subnet to be able to access the private subnet, which is confirmed via psql
on the EC2 command line after ssh, but failing from within a python script after ssh or in a jupyter notebook hosted on the EC2 (behind Caddy reverse proxy), but also not working with ssh tunneling into the ec2 from local. Should just work?
I also tried fixing the SELinux setting sudo setsebool -P httpd_can_network_connect_db=1
but that didn't change anything
I've also played around with very permissive security groups since this is a testing environment, still not working.