Create a New NPC

[Vincent]

Hello everyone, i'm new to the project, I tested pXI some years ago but it wasnt very advanced.
Just a few days ago i read about DSP and immediatly reinstalled FFXI to test, i was amazed, good job to everybody.
So i joined mvd's server and started searching a bit about how i could help the project. So far I've setup a server on my computer and am testing/learning about DS's architecture.

My motivation, as a player and a lover of videogames is to preserve FFXI for future generations. Let's say in 10, maybe 20 years (when SE's server will be down) someone wants to discover this wonderful game, if nothing's done now it will be harder later, just like those guys in the 80s dumped Arcade games so i could discover them.

Ok here are my questions :

What would be the process to create, not simply re-use, new characters/weapons/items/zones/dialogs ?
The way i see it, there are DAT files that contain all the raw infos, then the SQL tables and LUA files are here to link them together.
Lets say there is text A, text B, name X, number Y for a NPC in a DAT. When the player enters a zone NPC is called via "packets" ? Then its position/model is loaded via the SQL. Then the lua defines what actions are played and what triggers them, like : if clic > then print A andif Y=0 then print B etc etc.
So can we add new DAT files with new NPC dialogs ? Then just link the id in the SQL and LUA ?

Now what i couldnt clearly read anywhere was where is exactly everything. For example what defines the name of a NPC ? The data in the DAT ? the name in the DB ? Where are those lines of dialog stored ? I can see some in text.lua but not all. Some infos seem to overlap and it very confusing.

If it's already been explained somewhere i'm sorry about asking again, please feel free send me a link
Sorry for the long post, and thanks for the answers

[whasf]

What you're asking is way outside the scope of the project. Why do you want to create new stuff when we don't have everything working that is in the game already??

[Vincent]

For debug purpose, so not every single server changes differents npcs to gives maps gils and w/e, but everyone hs that new npc they can mess with at will. The original idea behibd tgis was to create a treasure hunting Npc, like marks in ffxii, you activate a hunt and you have certain conditions to fill for the NM/HNM to pop, this was solely for solo/duo purpose where respawn times condition should be leagues away from what they are for a MMO

[Vincent]

Ok ok even with no help I could manage to advance on this problem, and i'm halfway there ! Since I think this project can not go far without sharing I will give the community my findings.

So, first thing was to find the NPC list of the Zone, this is easy with POLutils.
In my case the area is Bastok Markets, the DAT file is ROM/27/44.DAT
There i had to understand how it was written



The first set of numbers is the name of the NPC in hex, pretty straightforward
The second set of numbers is the NPC' ID, but you have to write it backward (wtf, cant remember how i even found this) so it's 010EB004 which is 17739780 in decimal, and is the ID of Malene in the SQL table npc_list
The third set is the NPC i created, there were some IDs with empty NPC names so i just put it here, but you could insert more strings at the end i guess.

So, now the game knows there is a npc called Vay Hem with ID = 010EB0B4 or 17739956


then I added

Code: Select all

INSERT INTO `npc_list` VALUES ('17739956', 'Vay_Hem', '1', '-200.003', '-9.998', '0.956', '27', '40', '40', '0', '0', '0', '0', '27', 0x01000A0100100520163005400750006000700000, '32', '235');

in the npc_lsit.sql

'-200.003', '-9.998', '0.956' is the pos of the npc, i just typed @where in the game to check my position.
'27', '40', '40', '0', '0', '0', '0', '27', i have no fucking clue what this does, just copied it from another non-quest NPC.
0x01000A0100100520163005400750006000700000 is the look of the NPC, i havent had time to decrypt it yet, but it's pretty sure it's the HEX code for Race/Gender/Face/Gear, in this case it's a hume male with dreadlocks and dino jerkin set.
'32', '235' dont know what it does.

Next I created Vay_Hem.lua in \scripts\zones\Bastok_Markets\npcs and pasted some other NPC script in it to see if it worked.

IT WORKS so i have that totally new NPC that I can target and trigger.
Now to make him talk something custom, my first headhache of the day

I had to find the Dialogs DAT, samething than with the NPC dat, use POLutils and it's in ROM/25/44.DAT
Here comes the twist, you can read it in POLutils but when you open it with an HEX editor it's all messed up.

That's when i tumbled on that thread http://polutils.7.forumer.com/viewtopic.php?t=147 and I had no bloody clue what the whole thing was about XORed ? 0x80 ?

Nontheless i'm adventurous lol so I decided to first see if I could encrypt myself a word jsut like FFXI does. A way to check if my encoding was right was if i cuold match my result with what's in the DAT, so i had to translate a word said by some NPC. But if the word was too common i would have had too many results.
I then spoke with a crapload of NPC to see if they said some special word. It happened to be Aquillina, she uses the word "stepfather", and it's the only occurence of the word in Bastok Market. To find that I exported the dialogs dat (25/44.DAT) to xml via POLutils, then i could search the word.

I used that table http://unicode.org/Public/MAPPINGS/VEND ... /CP932.TXT to type "stepfather" in hex. Then applyed a XOR of 80808080808080 (or whatever the fuck that is) with that wonderful tool http://www.darkfader.net/toolbox/convert/ and it gave me a string of hex code.

Then i did a search for this string in the dialogs' DAT and fount the exact same one.

I then converted some custom text the same way, encrypted it and replaced the word stepfather.

Next thing i had to do was replace the EVENT ID of Vay_Hem.lua to be the same as Aquillina so when i clic my npc it says her sentence.

and TADAAA



Now there are two more problems,
How do you tell the game that it's the CUSTOM NPC that is the ACTOR of the EVENT instead of the other NPC ?
I dont know if ACTORS are directly coded in the CS, in this case it's very very difficult to do, or if they are CALLED via another event, like use THAT actor with THAT event, in which case we would still keep the event but just change its actor.

How could you script your own EVENTs ? I dont know how these are encrypted yet.


One more thing, the process of converting the DAT's encrypted HEX into readable TEXT then re-encrypt the new text is very long to do manually, anyone with some coding knowledge could write a simple tool to READ/WRITE/ENCRYPT dialog DATs ?

By the way i'm a CG Artist so

[kjLotus]

interesting

xor'ing each byte with 0x80 is kind of odd

One more thing, the process of converting the DAT's encrypted HEX into readable TEXT then re-encrypt the new text is very long to do manually, anyone with some coding knowledge could write a simple tool to READ/WRITE/ENCRYPT dialog DATs ?

xor is its own inverse, so you can just xor the entire file with 0x80808080...(etc) to get readable text

would also be really simple to do in c if you want something to do :p

edit: damn you for sidetracking me! this is all of 25/44.dat converted looks like (it's pretty big)

[Vincent]

Wow thanks Lotus and sorry for sidetracking you
It's cool how you could do that so fast Oo i would need to spend hours learning C... Is it hard ? I dont know any prog language

I found the complete encryption keys for Text and non-text data, seems they are based on the data itself http://code.google.com/p/polutils/wiki/ ... Encryption

[Vincent]

Getting cloooser, apparently some guy tried to make a 3Dsmax > DAT python script (which is awesome cause I've been using 3dsmax every day for the past 15 years ^^) to export vertex/faces/bones/weights.

I'd try to contact him cause the post died a few months ago, but they have a paying registartion to their forum (wtf people...)

but they left a few clues nontheless so i'll post it for future reference

Anyways let's just suppose this is a chunk-based format, where it gives the tag along with 4 bytes that identify what kind of information is stored and maybe some counts

The first tag is always followed by 0x 01 01.
There will always be a corresponding "end" chunk somewhere later.

You can see that the texture chunks also some common bytes following it (0x20)

Having this parenthesis matching makes it easier since now you can just do a while loop and build a parse tree from it.

Everything appears to be 16-byte aligned (judging from the pixel data, unless those 00's at the end are common).

There appears to be a chunk size somewhere. For example if you look at shiryu, the "cse0" chunk has token 0x87.
If you look a little further, you'll see 0xD8 or 216. If you look at the size of the entire chunk, it is about that much.

Now look at the "mse0" chunk at offset 0x200. It also has token 0x87, but this time if you look at the chunk size, you see 0x144, or 324.

Again, if you highlight the rest of the bytes until you get to the next chunk ("mse1"), you'll see it is a little more than 324 bytes.

I would imagine all dat files have the same structure.
I looked at a couple that were dealing with things like UI, and they were stored in the same way.

Some dat files even have only 4 bytes: 00 00 00 00

The textures are pretty much the same things. You just have to see what pixel format it uses to calculate the size of the pixel data.

Hmm, so we have a bunch of entries at the top (maybe stuff like bones?), followed by a bunch of textures, followed by vertices, and then followed by faces (which contains mat name as a string, vert indices as shorts, and the UV's as 6 floats)

Another per-face format great...I can see why converting this to MQO was a natural choice just because when I'm looking at it all that comes to mind is MQO MQO MQO MQO MQO MQO

[whasf]

Maybe that could help us in extracting the 3D map too so we can work on collision detection. Right now monsters aren't aware of the terrain, all collision detection is done by the client.

[Vincent]

Alright ! the news of today,

since i have a major cold and my brain is in slowmotion I decided to learn C tomorow and just focus on that mysterious "look" entry in the npc_list table.
As i thought it defines the Race/Sex/Face/Gear of the NPC

before I start to explain let me give you a little laugh



and that's how my poor NPC looks at the end of the day,
I did about a hundred tests to figure out exactly how to interprete these hex string
They look like 0x010002073F1037203C3058403C50D86000700000 and are put in the "look" column of the entries in the table "npc_list"

Here goes, i wont go through the whole process of finding out because it was alot of trial and errors, mainely because i split the code by pairs at first.

Code: Select all

0x01000A01C200C200C200C200C200020003000000
0xAABBCCDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLL

Ok so, first line we've got an example of the look's hex string, second line i put the corresponding letters to split the code.

A/B : No clue what this is, I just think 0x0100 is for regular NPCs, maybe it's some kind of flag to tell if they are clickable, or inactive or pass through, 0x0200 and 0x0000 is usually for test npcs and npc with weird names like 1 3 or qm2.

C : That's the code for the FACE model, there are 16 player face. For unique NPCs like Cid or Ghilgamesh, they have unique IDs, I see you coming already "OMG i'm going to put Cid with a galkan body, full dancer AF and make him fish !" well no...




If you keep 0x0100 and then put 3F (Cid's hex face code) it wont do anything, (I think the game will loop the adress and look in the player heads, like a modulo, i could be wrong tho)
If you put 0x00003F00 well... it seems like it's loading the wrong/cant load the bones/weights.
Couldnt test if further, maybe later.

D : that's the code for the RACE. 08 Galka / 07 Mithra / 06 Taru F / 05 Taru M / 04 Elvan F / 03 Elvaan M / 02 Hume F / 01 Hume M / 00 Hume M or Unique NPCs

E : HEADGEAR
F : BODY
G : HANDS
H : PANTS
I : FEET
J : Main Hand
K : Off Hand
L : no clue, i have put like 50 different random values there, does squat.

I didnt note everything because I got invisible returns alot, these adresses dont seem to make any sense, except the higher you go in numbers the more you advance in time, like 0-128 Vanilla, 128-196 ZM, 196-272 CoP etc (those are made up numbers), but it's very hard to find the highest number, because the results seem to loop, or have a default value if you go too high (if you put FAF9 it might give you the Leather or Bone set)
But the decimal values dont match either Altana Viewer nor the IDs in the items_armor table. I suppose it's because NPC often have some custom outfits, I dont know.

Anyway the good news is that you can load any player model (including armor) in any zone, because they are loaded. So you can have Abyssea armors on npcs in Valkurm without problem. Another detail, the hex matches from slot to slot, meaning if you put a value in the head slot, and the same in the other, they will be from the same set (thank god).

Here are some examples

0100 Leather
1000 Raptor
2000 Adaman
C400 Denali

One more thing, in the npc_list table there is a column named animation, i toyed a bit with it and 1 battle / 3 death / 5 chocobo / 6 fish / 8-9 invis / 13 idle, got tired then.


If anyone knows how to get a full list of the hex codes for the armors/weapons it would be a grandiose help I will make a big tutorial for the wiki once all of this is clear.

[whasf]

I hope Diatanato chimes in, he knows this stuff pretty well