Can anyone help me understand what tools/software already exist that could help me in building a residential proxy network? I have access to residential nodes (say 10-20) and I want to connect them to some public API/gateway such that a client can make a single HTTP/S request to that gateway and have it route through one of the residential nodes. Things to consider:
* Residential nodes are behind routes/NAT so they can’t expose ports publicly.
* The gateway would have to be hosted somewhere e.g AWS. Maybe there’s already a commercial service that allows me to connect my own nodes to it? And it just routes traffic to those nodes.
My goal: Looking to significantly reduce the costs of routing traffic through residential proxies ( running/owning the nodes myself is the best way to do that). Also just curious to understand ways to implement this.
You already have a residential network and it is behind a NAT. So you do not actually need a proxy to allow "a client can make a single HTTP/S request to that gateway and have it route through one of the residential nodes. ". You can just configure your NAT router to enable port forwarding and forward certain ports from the public internet to your residential nodes.
As mentioned by others, another option is tunneling such as ngrok or https://pinggy.io/ .
One command can be run in a residential node to share a port such as:
ssh -p 443 -R0:localhost:8000 a.pinggy.io
you could create a VPN where all devices are connected to. We use zerotier
BTW just found Scrapoxy io possibly be the central control and has other features which you might find useful…
Not sure if there is something dedicated but you can use squid proxy for both
One config for the nodes to pass traffic
And one for the central node, you can use different ports for different nodes as a simple setup, can probably get mode advance
Thanks yeah familiar with ngrok it’s an option.
Zerotier looks cool. I’ll check it out thanks!
Sounds like idea would be for all the residential nodes and gateway service to run the VPN agent to make them all accessible on the same virtual network. That way when the gateway needs to route a request to a proxy, it can just forward it to some IP:Port on the virtual network.
Scrappy io? I can’t find it.
Thanks for the response.
How do I connect central node and residential proxies though? The residential proxies will all be behind home routers and therefore can’t (easily) expose the ports the squid server is listening on… and therefore the central node can’t reach out and connect/forward requests to the residential proxy.
So my thinking is the residential nodes will need to make outbound persistent connections to central and I don’t think that’s something squid supports out the box?
scrapoxy.io
Sorry wanted to reference this I think the phone fixed spelling
There are some services that make it easy to put your node easily accessible used in software testing, I can’t recall a name right now, some kind of postman helper
Look up tunneling I guess, maybe look up ngrok
Got it thanks. I checked it out and looks good although more geared towards providing the ‘gateway’ portion of the stack I described above. I don’t *think* scrapoxy provides a solution for the NAT traversal issue / residential proxy nodes.
However, reading the resources on scrapoxy.io I came across proxidize.com. That seems a lot closer to what I’m looking for. They provide a solution for turning mobile devices (especially Android phones) into mobile proxies that you route requests through with the help of a cloud gateway. But the proxy nodes don’t have to be android devices (they offer a linux binary you can download) and I don’t see why you couldn’t just run that binary on a computer/raspberry pi in a home. Looks promising!
If you can tunnel into the master node, doesn’t matter what software, the control node could list the possible tunnels as proxies using localhost+port. This gateway can check their health ect and use what’s available and use only them