Python AH Buyer / Seller / ffxiah.com scrubber

[pureluck]

Question, and excuse me if this has been asked already. I went through all 13 pages quickly and didn't see anything. But how do i generate/scrub the prices to an specific server, like say "Asura". Because I want to model towards those prices?

[Lucious_Llama]

Good question. I wrote it so long ago I don’t remember at this point. I think it is defaulted to Bahamut because the default.

Well, looking at the code and fooling with ffxiah a bit, it seems data is coming from urls of the form https://www.ffxiah.com/item/4096, where 4096 is the item number (fire crystals in this example).

Most importantly, when you choose a different server from the drop down menu on the left, the url does not change..

This suggests to me that the server chosen is handled by JavaScript calls, which can only be executed by a browser like Chrome or Internet Explorer. Unfortunately my script does not use a browser, and so there is no way for it to execute JavaScript inside the html text it downloads. So... Without fundamentally changing how the scrubber script downloads data, I don’t think there’s any way to choose the server. If someone figures out a url that requests item data for a specific server, it could be added.

More generally, scrubbing websites that rely on JavaScript is a thing I am not sure how to do without using something (a browser) that executes the JavaScript for you. That can be done in python with the selenium module, for example, but feels extremely hack like and broken to me. I’d definitely be interested in anyone that knows how to do this. My other thoughts would be to use a framework like Qt to execute JavaScript upon getting the html text.

[ClanMcCracken]

Does beautifulsoup revert python back to 2.7? I downloaded anaconda 3.6, installed 3.6, tested 3.6 did all all of the updates, and checked my version again, and im back to 2.7

[Lucious_Llama]

It does not.

I really do think you still have some python2.7 somewhere installed if that’s happening. I’m not sure what to tell you without knowing more information. Here is my best guess on what to do:

1) Search your entire computer for python.exe, and determine if there’s more than one.

2) If you really have a python.exe that is 2.7, I think you should edit your path environment variable as described here: https://www.computerhope.com/issues/ch000549.htm

Remove anything that says the word python in it. It is a semicolon separated list. So if it’s blah/blah;some/path/to/a/python;/foo/bar/baz change it to blah/blah;/foo/bar/baz

[tarumancer]

I don’t think there’s any way to choose the server. If someone figures out a url that requests item data for a specific server, it could be added.

add this to the URL:

?sid=x (e.g. https://www.ffxiah.com/item/4096/fire-crystal?sid=28)

x=

1=Bahamut
2=Shiva
5=Phoenix
6=Carbuncle
7=Fenrir
8=Slyph
9=Valefor
11=Leviathan
12=Odin
15=Quetzalcoatl
17=Siren
20=Ragnarok
23=Cerberus
25=Bismarck
27=Lakshmi
28=Asura

Seems to only work on fully qualified URLs (not by item ID).

edit: Also, it's a cookie value, so if you can define/pass a cookie, you can probably specify it that way.

site: ffxiah.com
Cookie Name: sid
Content: <numeric value>

[Lunitari]

I can confirm that after following the steps it downgrades your python to 2.7. Can anyone help?

[Delaide]

It is the beautiful-soup, it pulls downgrades.
Instead of using

Code: Select all

conda install beautiful-soup

use

Code: Select all

conda install beautifulsoup4

This will add beautiful soup without downgrading the python build. Tested (as I had the same issue) and works.

OP, can you check your documents on the website and update?

[Lunitari]

IT WORKS! Thank you so much!
small question- where can i edit to select the server, as mentioned above?

[Delaide]

Sorry, no idea about that...

[xstealth]

I know this thread has been slow for a while, but I'm using this on my local private server and really enjoy it, but would love if a few more options were added. Specifically a way to set a percentage chance of your item being bought by the script. As of now, unless I'm missing something, it buys everything listed by players 100% of the time as long as your price is the same or lower than that the program lists them for. I'd love an option to manually set a percentage chance in the config.yaml file so I could change it to, say... a 25% chance of sale or something on a per item basis to add a bit more realistic feel to the economy. Thanks!