Ubuntu Server 13.04 Setup - x86 & x64

[sanon]

Build notes for the most current version of Ubuntu Server 13.04 64-bit & 32-bit

I'm posting this information in the hopes that it may benefit the community. Please let me know if you see anything terribly wrong.

• Performed standard server install with SSH & LAMP options.
  (I know LAMP was not necessary just for MySQL. I may revise this at a later date)
  Added user ffxi.

Log in as ffxi.
Set root password:

Code: Select all

sudo passwd

Logout.

Log in as root. (I did and I do. Please be aware that if you are logging in as a standard user that you must utilize 'sudo' where appropriate.)
Initial system updates:

Code: Select all

apt-get update
apt-get upgrade
reboot

For 64-bit server:
Install tools and libs for 32-bit build environment:

Code: Select all

apt-get -fy install automake cpp:i386 g++:i386 gawk gcc:i386 liblua5.1-0-dev liblua5.1-0-dev:i386 libmysqlclient-dev:i386 libtool:i386 subversion

For 32-bit server:
Install tools and libs for build environment:

Code: Select all

apt-get -fy install automake cpp g++ gawk gcc liblua5.1-0-dev libmysqlclient-dev libtool subversion

Fix liblua.so linking error:

Code: Select all

cd /usr/lib/i386-linux-gnu/
ln -s liblua5.1.so liblua.so

Checkout SVN source:

Code: Select all

svn co http://onetimexi.googlecode.com/svn/trunk /home/ffxi/dsp

No longer required (thx lautan):

Download Linux dssearch patch here: (Can we get this posted to the GoogleCode downloads?)
download/file.php?id=820

Build:

Code: Select all

cd /home/ffxi/dsp
chmod a+x autogen.sh
./autogen.sh
./configure LIBS=-lpthread
make

Assuming that you've done as I had and installed the LAMP package on install, you have already set the MySQL password for the root user account so you can now login and create the database and user.

Code: Select all

mysql -u root -p
mysql> CREATE DATABASE dspdb;
mysql> GRANT ALL PRIVILEGES ON dspdb.* TO dspdb_user@localhost IDENTIFIED BY 'dspdb_password';
mysql> quit

Import SQL tables:

Code: Select all

cd /home/ffxi/dsp/sql
for f in *.sql; do echo -n "Importing $f into the database..."; mysql dspdb -u dspdb_user -pdspdb_password < $f && echo "Success"; done

Please be aware that there is intentionally no space between the "-p" switch and the dspdb_user's password.

Update your configuration files @ /home/ffxi/dsp/conf

Logout.

Login as ffxi.
At this point you can use 'screen' to background /home/ffxi/dsp/dsconnect, /home/ffxi/dsp/dsgame, and /home/ffxi/dsp/dssearch or you can use Ubuntu's "Upstart" configuration files to run each at startup.
Undecipherable data here: http://upstart.ubuntu.com/getting-started.html
'screen' here:http://linux.die.net/man/1/screen

I am currently using four (4) upstart configs, three (3) are standard standard upstart scripts with no start conditions (one each for dsconnect, dsgame, dssearch), and one (1) config is configured as a prestart script:

Example of my dsconnect upstart script:

Code: Select all

 nano /etc/init/dsconnect.conf

Contents of the config file:

Code: Select all

root@ffxi:/home/ffxi/dsp# cat /etc/init/dsconnect.conf
# DSConnect

description     "Start DS Lobby Login Server"

stop on stopping dsp

script
        cd /home/ffxi/dsp
        ./dsconnect
end script

respawn

dsgame & dssearch are configured through similar scripts.

My pre-start upstart script:

Code: Select all

root@ffxi:/home/ffxi/dsp# cat /etc/init/dsp.conf
# DSP FFXI Server...

description     "Start DSP FFXI Server"

start on started networking
stop on stopping networking
stop on starting shutdown

pre-start script
      start dsconnect
      start dsgame
      start dssearch
end script

This script is set to start all 3 services after networking has been established, or you may use:

Code: Select all

start dsp -or- restart dsp -or- restart dgame

Please note that, in this configuration, you do not want to configure the pre-start script for respawn as each service will manage that on it's own.

Additionally, for MySQL management install phpmyadmin:

Code: Select all

apt-get install phpmyadmin

At some point during the package install it will ask to install to lighttp or to apache. Since I had already installed apache (with LAMP package selection on install) I chose apache. I expect that configuring to lighttp may be an infinitesimally leaner server (if you hadn't already selected the LAMP package; if you had, fortheloveofallthat'sholy pick apache). After install, the admin page is available @ http://yerserveraddress/phpmyadmin, login with MySQL credentials.

That's all I've got at the moment.
-sanon

[lautan]

Thats some great stuff. I have some cool scripts that'll update the server and such. I can share if you're interested.

[sanon]

Thats some great stuff. I have some cool scripts that'll update the server and such. I can share if you're interested.

Thank you, I would be thrilled if anyone wanted to share their knowledge with me. I haven't even begun to consider how I might perform upgrades under linux with SVN, but that is something that I will want to address soon. Whatever you'd care to share I would be most appreciative.

Assorted linux related thoughts I'd had:

• MySQL DB backup strategies, replication, flat-file, etc?
  MySQL table diff'ing on svn updates? I'm assuming most people don't blow away their char_vars and AH on sql svn changes...
  Configs, settings.lua, src mod diff'ing. Also assuming most people wouldn't reset all modded EXP rates, etc.
  Better Ubuntu Upstart scripts with rotated logs so's I can 'tail -f' while I'm attempting to chat with some stone-faced npc.
  Inspect that dssearch.diff file and inquire as to why that may not be in the svn source.
  Additionally, I'd like to find\build a linux based AHbot service similar to: viewtopic.php/?f=16&t=19 or viewtopic.php?f=16&t=1116

I'm curious, is it fair to assume that most are running the server on Windows boxen?

Again, thank you for any help you may be able to provide.
-sanon

[bluekirby0]

Of the core developers, only lautan and I work with Linux at all, and only lautan uses it as his primary development platform. I think it is also fairly safe to assume more servers out in the wild are running on windows boxes, although if you are trying to set up a server on a VPS a linux box will cut your resource requirements immensely.

For database backups there are several strategies you can use, but I haven't messed with any myself since as a developer I don't really mind blowing away my entire database frequently.

More friendly upgrade options are something that can be looked at in the long-term, but I wouldn't personally devote the resources to it at this stage of the project.

The search server diff basically does a direct replacement of network functions that are windows-only with ones that are anything-other-than-windows-compatible using pre-processor macros. While they technically don't break windows compiles in any way, the guy who technically owns this project disagreed with my approach to it so I've kept it separate as a patch until the day comes when someone feels like rewriting the highly stable network code (as you can guess, its kind of silly IMO to rewrite it when the patch works flawlessly).

AH bots are fairly easy to build...just pick your coding language of choice, find a mysql interface for it, and set up your queries. All most of them do is inject pre-defined info into the auction house's table. If you wanted to get fancy you could scrape info from ffxiah.com and base prices off of actual retail sales history.

[sanon]

Of the core developers, only lautan and I work with Linux at all, and only lautan uses it as his primary development platform. I think it is also fairly safe to assume more servers out in the wild are running on windows boxes, although if you are trying to set up a server on a VPS a linux box will cut your resource requirements immensely.

For database backups there are several strategies you can use, but I haven't messed with any myself since as a developer I don't really mind blowing away my entire database frequently.

More friendly upgrade options are something that can be looked at in the long-term, but I wouldn't personally devote the resources to it at this stage of the project.

The search server diff basically does a direct replacement of network functions that are windows-only with ones that are anything-other-than-windows-compatible using pre-processor macros. While they technically don't break windows compiles in any way, the guy who technically owns this project disagreed with my approach to it so I've kept it separate as a patch until the day comes when someone feels like rewriting the highly stable network code (as you can guess, its kind of silly IMO to rewrite it when the patch works flawlessly).

AH bots are fairly easy to build...just pick your coding language of choice, find a mysql interface for it, and set up your queries. All most of them do is inject pre-defined info into the auction house's table. If you wanted to get fancy you could scrape info from ffxiah.com and base prices off of actual retail sales history.

Thank you for taking the time to provide me with a good set of information and background on the project. I would agree, that given that the code builds and is stable under Linux, that would certainly be the most resource efficient approach. It all boils down to user comfort level though, and the majority of the world isn't comfortable at a shell prompt. I get that.

Regarding the dssearch code, I did notice that the diff inserted platform-specific conditionals to determine whether or not I was going to use a winsock2 header. I'll admit that prior I had spent about 30 minutes googling how I might port the code over to linux before I did manage to locate the v7 patch file. To me, the patch looked like smart code, which is why I was unsure as to why it was not part of the base. However, I appreciate your saving me the headache of reinventing linux sockets.

For the AH, I did just now stumble upon a linux PHP script here: viewtopic.php?f=16&t=266 that I will review tomorrow. And I know that it would be a fairly simple SQL insert, but yeah, I wanted to get fancy and buy player sold items, restock, and price per FFXIAH, etc. Scraping isn't a strong suit of mine, but I may look at some of the Ruby/nokogiri gem functions, be as good an excuse as any to learn. Been a while since I looked at lua (probably my lost WOW server) so I'm excited about that at the moment as well.

Anyway, I do look forward to contributions to the platform from you both, and thank you both for allowing me to relive some of my past-life experiences in FFXI.

[lautan]

Code: Select all

#!/bin/bash

for file in `find sql -name '*.sql' -mtime 0`
# for file in `find sql -name '*.sql'`
do
	if [[ "$file" != *char* ]]
	then
   	    	echo "Migrating $file"
	 	cat $file | mysql -u dspdb -p dspdb --password=ffxi
	fi 
done

echo "update zone_settings set zoneip = '642354';" | mysql dspdb -u dspdb --password=ffxi

I use this script to migrate the db after svn update. It will find all changed sql files and run them. Opptionally you can ignore char tables.

[sanon]

Code: Select all

#!/bin/bash

for file in `find sql -name '*.sql' -mtime 0`
# for file in `find sql -name '*.sql'`
do
	if [[ "$file" != *char* ]]
	then
   	    	echo "Migrating $file"
	 	cat $file | mysql -u dspdb -p dspdb --password=ffxi
	fi 
done

echo "update zone_settings set zoneip = '642354';" | mysql dspdb -u dspdb --password=ffxi

I use this script to migrate the db after svn update. It will find all changed sql files and run them. Opptionally you can ignore char tables.

Excellent, thank you so much for this script. Everything about a semi-automated updater seems fairly simple, with the exception of the databases. Namely char's and accounts.

I had considered something along the lines of:

• Keep a separate trunk version DB with no acct/char data
  Apply all sql updates to TrunkDB
  Use mysqldiff or mysqldbcompare to generate a diff between blank TrunkDB and the production DB???
  Apply diff to Prod. (Profit!)

Edit: Attached notes that I had made regarding my masterplan. To be continued...

Code: Select all

Update SQL svn:
cd /home/ffxi/dsp
svn up sql

Truncate (Empty) SVN DB:
mysqladmin -f -u root -p{SN4RF} drop dspdb_svn
mysqladmin -f -u root -p{SN4RF} create dspdb_svn

Upload to SVN DB:
cd /home/ffxi/dsp/sql/
for f in *.sql; do echo -n "Importing $f into the database..."; mysql dspdb_svn -u dsp_user -p{SN4RF} < $f && echo "Success"; done

Dump DBs:
cd ..
mysqldump -u root -p{SN4RF} dspdb_svn > dspdb_svn.sql
mysqldump -u root -p{SN4RF} dspdb > dspdb.sql

Diff dumps:
diff -u --from-file=dspdb_svn.sql dspdb.sql > sqldiff1.diff

-=-=-= Source Update / Rebuild -=-=-=

update dsp:
cd /home/ffxi/dsp
svn diff > patches/svn/`svn info | grep -oP "(?<=Revision: )[^ ]*"`_`date +"%Y%m%d_%H%M%S"`.diff
svn revert . --recursive
>>TODO: Interactively apply most recent diff. 

FWIW:

--list all diff'd files:
svn diff | grep -oP "(?<=Index: )[^ ]*"

--list all diff'd files with revision:
svn diff | grep -oP "(?<=--- )[^ ]*.*"

I ran out of time at that point, but I would welcome any suggestions that you or the members may have regarding.
Thanks!
-sanon

[sython]

Hi, thanks for this tutorial. I get an error on the make...

g++ -std=gnu++0x -DHAVE_CONFIG_H -I. -I./src/common -DdsTCPSERV -g -O2 -MT dsconnect-lobby.o -MD -MP -MF .deps/dsconnect-lobby.Tpo -c -o dsconnect-lobby.o `test -f 'src/login/lobby.cpp' || echo './'`src/login/lobby.cpp
src/login/lobby.cpp: In function âint32 lobbydata_parse(int32)â:
src/login/lobby.cpp:111:87: error: inconsistent user-defined literal suffixes âCL_WHITEâ and âCL_RESETâ in string literal
src/login/lobby.cpp:111:87: error: unable to find string literal operator âoperator"" CL_WHITEâ
make: *** [dsconnect-lobby.o] Error 1


any suggestions?

[bluekirby0]

Odd, that error usually occurs when the architecture's pointer size exceeds the size of the platform's int type, i.e. when trying to store a 64-bit program's pointer in a 32-bit int which is a common problem on x86-64 systems. Granted, I'd just as soon the code be portable so this becomes a non-issue, but these inherent problems are ingrained rather deeply in the code.

[atom0s]

if (RBUFB(buff,0) == 0x0d) ShowDebug(CL_RED"Posible Crash Attempt from IP: "CL_WHITE"<%s>\n"CL_RESET,ip2str(session[fd]->client_addr,NULL),NULL);

There are missing spaces between the CL_ tags and the rest of the content.