403 Client Error - ReserveCalifornia blocked on cloud hosting providers

[DavydLiu]

Describe the bug
I got the following error when I run the below command from a linode machine:

HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters

Original Camply Command (with --debug)

camply campsites --provider ReserveCalifornia --campground 597 --campground 598 --start-date 2024-02-10 --end-date 2024-02-11 --search-forever  --notifications pushover --polling-interval 5 --debug

Expected behavior
Running the same command on my laptop works fine

Console Output (with --debug)

[2024-01-31 02:00:45] CAMPLY camply, the campsite finder ⛺️
[2024-01-31 02:00:45] DEBUG Setting up camply debugging
[2024-01-31 02:00:45] DEBUG Camply Version: 0.32.1
[2024-01-31 02:00:45] DEBUG Python Version: 3.11.6
[2024-01-31 02:00:45] DEBUG Platform: linux
[2024-01-31 02:00:45] INFO Using Camply Provider: "ReserveCalifornia"
[2024-01-31 02:00:45] INFO 1 booking nights selected for search, ranging from 2024-02-10 to 2024-02-10
[2024-01-31 02:00:45] DEBUG Refreshing UseDirect Metadata: filters.json
[2024-01-31 02:00:45] DEBUG Starting new HTTPS connection (1): calirdr.usedirect.com:443
[2024-01-31 02:00:45] DEBUG https://calirdr.usedirect.com:443 "GET /rdr/rdr/search/filters HTTP/1.1" 403 520
[2024-01-31 02:00:45] CAMPLY Exiting camply 👋

Additional context

[juftin]

Oh no! I'll take a look. Hopefully ReserveCalifornia isn't cracking down on tools like camply

[juftin]

EDIT: I'm seeing that this is only occurring when coming from linode servers and not from local. The command from above worked for me as well.

This might just be an unfortunate blocking of linode IP addresses by the ReserveCalifornia servers. The best I can do in this case is update camply's user agents to a more recent version of fake-useragent

[DavydLiu]

I actually tried Linode and DigitalOcean, both had the same issue.

[DavydLiu]

Tried vultr this morning, still the same error.

[juftin]

😞 I've updated fake-useragent to the latest version and enforced a random User-Agent for each request to ReserveCalifornia. Sadly this is the best I think I'll be able to do - I wouldn't be surprised if ReserveCalifornia universally blocks all major cloud hosting provider's IP ranges.

Let me know if this resolves the issue for you - if not you may want to consider a VPN or similar solution to mask your IP address.

[DavydLiu]

Thanks, let me git it a try. I'm also curious which cloud provider you'd recommend for hosting camply?

[juftin]

I don't have a recommendation sadly - my cloud provider is a small linux server running in my basement that I also use for my homelab. A raspberry pi would probably also work nicely.

This is actually the first I've heard of this cloud provider blocking issue so you'd probably only run into this problem on ReserveCalifornia and not the other camply providers (for now).

[DavydLiu]

Just tried 0.32.2. Unfortunately, it's still blocked for HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters. Recreation.gov works perfectly. I'm curiously what ReserveCalifornia does differently.

Raspberry PI will be my last resort.

[justin-outdoorithm]

I had this issue on Digital Ocean and ended up using a proxy

[DavydLiu]

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

[justin-outdoorithm]

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

[DavydLiu]

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy.

[justin-outdoorithm]

I've been using the Residential Pay As You Go plan

…

On Mon, Apr 1, 2024 at 10:12 AM David Liu ***@***.***> wrote: Smartproxy I had this issue on Digital Ocean and ended up using a proxy Which proxy did you use? Smartproxy Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy. — Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCRKPZBP5OIRUD4TOY7Z6XDY3GIPJAVCNFSM6AAAAABCSIENPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGE3TONZVGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[botlahan]

I've been using the Residential Pay As You Go plan
…
On Mon, Apr 1, 2024 at 10:12 AM David Liu @.> wrote: Smartproxy I had this issue on Digital Ocean and ended up using a proxy Which proxy did you use? Smartproxy Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy. — Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCRKPZBP5OIRUD4TOY7Z6XDY3GIPJAVCNFSM6AAAAABCSIENPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGE3TONZVGQ . You are receiving this because you commented.Message ID: @.>

I’m having the same issue while trying to use camply on a Google cloud virtual console. Any idea on how to setup a third party residential proxy such as smartproxy on Google cloud compute engines?

[blkwtkns]

Before I found your project, I had made my own node app to basically do the same stuff, except only for reserveCalifornia, though it's no where near as mature as this awesome project. Thanks for this!

Okay anyways, I had to setup a proxy for running my script, because when I would run it without one, it would give 400/500 type of codes after running once or twice. AND, I wouldn't be able to look through the reserve california site for at least 3+ minutes. So I was finding myself in this box of finding reservations, but not being able to act on them fast enough.

So yeah, I configured a proxy, and it's been mostly good. Until today. something changed today, and my requests are failing hard. So then I started looking around and found your project. I really like it.

So my question in relation to this issue right here - would it be possible to allow users to integrate a proxy into this, instead of just relying on the fake useragent stuff? Thanks!