Python AH Buyer / Seller / ffxiah.com scrubber

Applications/Tools written to make running DarkStar easier for the rest of us.
pureluck
Posts: 3
Joined: Fri Mar 09, 2018 4:48 pm

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by pureluck » Fri Apr 06, 2018 12:11 am

Question, and excuse me if this has been asked already. I went through all 13 pages quickly and didn't see anything. But how do i generate/scrub the prices to an specific server, like say "Asura". Because I want to model towards those prices?

Lucious_Llama
Posts: 40
Joined: Mon Jul 28, 2014 5:22 pm

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Lucious_Llama » Fri Apr 06, 2018 6:25 am

Good question. I wrote it so long ago I don’t remember at this point. I think it is defaulted to Bahamut because the default.

Well, looking at the code and fooling with ffxiah a bit, it seems data is coming from urls of the form https://www.ffxiah.com/item/4096, where 4096 is the item number (fire crystals in this example).

Most importantly, when you choose a different server from the drop down menu on the left, the url does not change..

This suggests to me that the server chosen is handled by JavaScript calls, which can only be executed by a browser like Chrome or Internet Explorer. Unfortunately my script does not use a browser, and so there is no way for it to execute JavaScript inside the html text it downloads. So... Without fundamentally changing how the scrubber script downloads data, I don’t think there’s any way to choose the server. If someone figures out a url that requests item data for a specific server, it could be added.

More generally, scrubbing websites that rely on JavaScript is a thing I am not sure how to do without using something (a browser) that executes the JavaScript for you. That can be done in python with the selenium module, for example, but feels extremely hack like and broken to me. I’d definitely be interested in anyone that knows how to do this. My other thoughts would be to use a framework like Qt to execute JavaScript upon getting the html text.

ClanMcCracken
Posts: 26
Joined: Sun Mar 25, 2018 8:54 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by ClanMcCracken » Sun Apr 08, 2018 4:38 pm

Does beautifulsoup revert python back to 2.7? I downloaded anaconda 3.6, installed 3.6, tested 3.6 did all all of the updates, and checked my version again, and im back to 2.7

Lucious_Llama
Posts: 40
Joined: Mon Jul 28, 2014 5:22 pm

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Lucious_Llama » Sun Apr 08, 2018 9:15 pm

It does not.

I really do think you still have some python2.7 somewhere installed if that’s happening. I’m not sure what to tell you without knowing more information. Here is my best guess on what to do:

1) Search your entire computer for python.exe, and determine if there’s more than one.

2) If you really have a python.exe that is 2.7, I think you should edit your path environment variable as described here: https://www.computerhope.com/issues/ch000549.htm

Remove anything that says the word python in it. It is a semicolon separated list. So if it’s blah/blah;some/path/to/a/python;/foo/bar/baz change it to blah/blah;/foo/bar/baz

tarumancer
Posts: 5
Joined: Thu Nov 12, 2015 10:30 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by tarumancer » Tue Apr 10, 2018 10:19 am

Lucious_Llama wrote:I don’t think there’s any way to choose the server. If someone figures out a url that requests item data for a specific server, it could be added.
add this to the URL:

?sid=x (e.g. https://www.ffxiah.com/item/4096/fire-crystal?sid=28)

x=

1=Bahamut
2=Shiva
5=Phoenix
6=Carbuncle
7=Fenrir
8=Slyph
9=Valefor
11=Leviathan
12=Odin
15=Quetzalcoatl
17=Siren
20=Ragnarok
23=Cerberus
25=Bismarck
27=Lakshmi
28=Asura

Seems to only work on fully qualified URLs (not by item ID).

edit: Also, it's a cookie value, so if you can define/pass a cookie, you can probably specify it that way.

site: ffxiah.com
Cookie Name: sid
Content: <numeric value>

Lunitari
Posts: 4
Joined: Mon May 14, 2018 9:52 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Lunitari » Thu May 24, 2018 10:31 am

I can confirm that after following the steps it downgrades your python to 2.7. Can anyone help?

Delaide
Posts: 445
Joined: Sat Jun 14, 2014 8:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Delaide » Thu May 24, 2018 10:12 pm

It is the beautiful-soup, it pulls downgrades.
Instead of using

Code: Select all

conda install beautiful-soup
use

Code: Select all

conda install beautifulsoup4
This will add beautiful soup without downgrading the python build. Tested (as I had the same issue) and works.

OP, can you check your documents on the website and update?

Lunitari
Posts: 4
Joined: Mon May 14, 2018 9:52 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Lunitari » Fri May 25, 2018 8:28 pm

IT WORKS! Thank you so much!
small question- where can i edit to select the server, as mentioned above?

Delaide
Posts: 445
Joined: Sat Jun 14, 2014 8:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Delaide » Mon May 28, 2018 4:52 am

Sorry, no idea about that...

Post Reply