how to create a scripting language?

Member
Posts: 304
Joined: 2002.04
Post: #1
I was wondering if anybody has some good resources on how to create a scripting language. I found this:

http://www.gamedev.net/reference/list.asp?categoryid=76

But any other resources or info would be appreciated.

I think interpreted would be easier to write - but compiling scripts to run in a virtual machine would run faster?

Also what features do you want in a game scripting language?

Please dont suggest I just use Spidermonkey or Ruby or whatever - I know there are great scripting languages out there already - I have reasons for wanting to write my own. Any other thoughts, resources, or suggestions are appreciated.

thanks,
Codemattic
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #2
Implementing an interpreted would be much easier. If speed is an issue, writting your own is a bad idea. Unless you get really into this you probably won't be able to write a VM that could run faster than most interpreted languages.
Probably the most important speed critical parts are going to be name lookups. Interning and hashing symbols efficiently is critical for this.

Everyone is going to ask why you want to do this though. Writing a good language is going to be a huge undertaking. Outdoing another language isn't going to happen unless you pour a lot of effort into it. It could be enjoyable if that's what you wanted to get out of it.

Features? A couple of collections, at least an array and hash type. Floating point and vector math are a must. Easy file acces is good. OO is always good. Advanced string manipulation can be handy but not crucial for a lot of games. Obviously you'd need access to game objects like sprites or models.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Sage
Posts: 1,403
Joined: 2005.07
Post: #3
Learn how to use function pointers in whatever language you plan to program your language in, it makes things a hell of a lot more modular.

in ruby you could do somthing like this to store commands:
Code:
# commands is a list of:
# [function name, number of parameters, code block to calls]
@commands = [
    ["dbg", 1, proc { |a|
        @cmd_error = "Matrix #{a.to_i} is of dimension (#{@tmp[a].rows}, #{@tmp[a].cols})"
        return false
    }],
    ["skp", 0, proc {
        @skp_prt = !@skp_prt
        return true
    }],
    # Creates a new matrix a by b in size, identity if a = b
    ["new", 2, proc { |a, b|
        @tmp[@tmp_pointer] = Matrix.new(a, b)
        return true
    }],
...
...

then writing a parse loop that will scan through the commands array and call the apropriate function(s) is easy.
Depending on how you want to deal with things like loops and conditionals, you might want to push blocks of code to be exected several times or you could move the cursor around the file and continue parsing.

Edit: in the way of resources look at the source code to lua
and if you want a simple example to take apart
asm_style_calculator.rb
Code:
@commands = [
    ["set", 1, proc { |a| @tmp=a }],
    ["add", 1, proc { |a| @tmp=@tmp+a }],
    ["sub", 1, proc { |a| @tmp=@tmp-a }],
    ["mul", 1, proc { |a| @tmp=@tmp*a }],
    ["rec", 0, proc { @tmp=1/@tmp }],
]

def find_cmd(name)
    0.upto @commands.length-1 do |cmd|
        if @commands[cmd][0] == name then
            return cmd
        end
    end
    return nil
end

@tmp = 0

line_number = 0
IO.foreach(ARGV[0]) { |line|
    line_number = line_number + 1
    
    if line.index(' ') != nil then
        cmd_name = line[0..line.index(' ')-1]
        cmd_rest = line[line.index(' ')+1..-1]
    else
        cmd_name = line.chomp
    end
    
    cmd = find_cmd(cmd_name)
    
    if cmd == nil then
        puts "Line #{line_number}: Unknown command: \"#{cmd_name}\""
        exit
    else
        if @commands[cmd][1] > 0 then
            if cmd_rest == nil then
                puts "Line #{line_number}: No arguments passed, #{@commands[cmd][1]} required"
                exit
            end
            cmd_args = cmd_rest.split.map { |s| s.to_f }
            if cmd_args.length != @commands[cmd][1] then
                puts "Line #{line_number}: 3 args passed, 4 needed"
                exit
            end
        else
            cmd_args = nil
        end
    end
    
    @commands[cmd][2].call(*cmd_args)
    puts @tmp
}

testfile.txt
Code:
set 3
add 2
rec
sub 1

Sir, e^iπ + 1 = 0, hence God exists; reply!
Quote this message in a reply
DoG
Moderator
Posts: 869
Joined: 2003.01
Post: #4
Unless you are at the level of a comp. sci. doctorate, you're not gonna be able to make a scripting language that can be called good.

You can write a scripting language, and the associated compiler/interpreter and/or virtual machine for educational purposes, if you are interested in the inner workings of such things. Don't expect it to perform well. Also, there are a lot of pitfalls you might never have thought of.

Now, if you could share the reasons why you want to roll your own, maybe we could give more advice than "don't do it".
Quote this message in a reply
Member
Posts: 245
Joined: 2005.11
Post: #5
Although not on quite the same scale, last summer I wrote a game engine for a text adventure. Since the idea was that I wanted to keep gameplay entirely separate from the engine while still providing as much flexibility as I could, so I needed to include a scripting system to set up the puzzles and so on. The syntax was quite basic and there was a wierd/useful/annoying glitch in the flow control statements (depending on what you were trying to do), but I got it working fairly well.
My approach was to compile scripts into a list of opcodes when I loaded the game data. This then made it fairly trivial to run the scripts when needed using a couple of pointers and a big switch statement. I'm sure it would have been more efficient to use a big array of function pointers or something but it was only a text adventure so speed wasn't a huge issue.
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #6
Tobi and I created our own scripting language, used for Antack, Dead Days, and Dawn of the Derby, and unfortunately major portions of the iGame3D application, in the end after three years and 135 commands, we switched over to Lua, which has books and websites already dedicated to it. My major goal in creating the script commands was four letter words that any three year old could read, now our Lua commands are these huge statements full of underscores and parathesis which I find unpleasant to read.

Tobi would be better able to explain how he achieved this, and if Antack or Dead Days source code is floating around somewhere (ie, as was promised in the uDevgames propaganda year after year) then it has the language in the T3DScript.h file. I tried to get people to adopt this crazy assembler looking stuff for years, didn't happen.

The major drawback to creating your own language is surely not speed as that was never an issue for us at all, it is documenting it and explaining to others how to use it, and then realising that the time they spend learning it is, in the end, a complete waste since they will need to learn a standardized and widely used language anyway in order to make themselves employable and to more
easily get others to help them in their game development.

I know you don't want to be told "use something already established", but if you can make your application compatible with things people already modding at the scripting level, it should surely extend the life of whatever you create.
Quote this message in a reply
Sage
Posts: 1,403
Joined: 2005.07
Post: #7
oops, I forgot to mention in my post. Dont do it! Rasp

Sir, e^iπ + 1 = 0, hence God exists; reply!
Quote this message in a reply
zKing
Unregistered
 
Post: #8
I'm not sure exactly what kind of help you are asking for, but ...

I've had to do some similar interpreters/compilers in the past. My biggest recommendation is to build a BNF description of the langauge on paper first so you have a clarity on what you want to do. Poor specification is not good with most software, but when building a compiler/interpreter... it's suicide. You can start with something small and add more to it later, but don't try to just wing it or you'll end up with a mess of spagetti code very quickly. You can use something like yacc/lex, which I've done, but in my opinion the effort to learn those tools doesn't pay off... they are a bit complicated to setup and it's VERY easy to forget how to use them so I personally think the the maintainance effort is too high.

One of the better concise descriptions of how to build a little interpreter that I've seen recently was in "The C++ Programming Language" (special edition?) by Stroustrup starting in Chapter 6 "Expressions and Statements". He builds a little calculator interpreter as his example. That said, I've got some experience building these things so his little example may be too concise for someone coming in cold.

I'll add the obligatory "this is not the most efficient use of time in 99% of cases"... but that said, it is _really_ fun to write an interpreter/compiler with your own hands. And it certainly makes you appreciate what the pro compiler guys have to deal with. Wink

If you have more specific questions please post them.
Quote this message in a reply
Member
Posts: 198
Joined: 2005.01
Post: #9
DoG Wrote:Unless you are at the level of a comp. sci. doctorate, you're not gonna be able to make a scripting language that can be called good.

Oh, come now, compilers and languages are not that hard, even decent ones. I'd bet that most of the "big" scripting languages right now were not written by doctorates. A basic class in compilers and automata theory at an undergrad level is enough to understand where you're going; that or a lot of independent research.

I've written several scripting languages for game projects earlier in the day, before there were a million to choose from. Two of them were compiled to byte code and executed on a virtual machine, and the second one of those was object oriented.

It still takes a while though. And a few tries before you figure out the ins and outs. And then usually you realize when you're done that you have a huge heap of non-standard, unsupported-by-anyone-but-you code. Smile Want bindings for some library? You get to write 'em! In something like Lua, Python, Perl, etc, there's a good chance that someone has already done a lot of work for you. That's why most people recommend not writing your own if you just want it for another project.

It can be fun to do in and of itself though.

Cryptic Allusion Games / Cryptic Allusion, LLC
http://www.cagames.com/
Quote this message in a reply
Moderator
Posts: 3,579
Joined: 2003.06
Post: #10
The standard warnings, blah blah blah: Like iGame3D pointed out, the biggest disadvantage to rolling your own is adoption by others. It's just so much easier to let other people consult tutorials and documentation for existing languages. I dislike the idea of making users learn an entirely new language that they'll never use with anything else, regardless of whether or not it is well done. I think it is much better to convince them to learn something like Lua or Ruby for your engine since they can use those skills later with other projects. It is a lot of work to make a compiled scripting language of your own, but it is most certainly not impossible. You might even do a good job of it!

Warnings aside, I can highly recommend Game Scripting Mastery by Alex Varanese. It's easily one of my favorite programming books. It starts off with how you could implement your own command based language and then moves on to building your own real deal, full-on compiled scripting language (which is what the book is really about). I got as far as making my own assembler before I finally realized how nice and easy Lua is. Implementing other scripting engines like Lua and Python are also covered in the book BTW. Of note, like most books, it is not written with the Mac in mind, but it isn't heavily Windows-centric either, so any experienced developer should have no problem with it on any system. It will explain to you how to make a fast, efficient, custom compiled scripting language of your own, from beginning to end. One of these days I will finish my own custom scripting language because it is actually kind of fun to do, but I think it's more of an academic excercise than anything else.
Quote this message in a reply
Member
Posts: 131
Joined: 2004.10
Post: #11
I don't entirely subscribe to the notion of not doing something. It's often the best way to understand the nuances of the problem instead of just plugging in a black box and pray that it works. After finding out and understanding the issues, plugging in the black box (lua, python, perl, whatever) will make more sense.

Starting out in making a scripting language will benefit from planning it out first. I've made a dead simple scripting language but Initially I made some false starts because I didn't really plan it out. But it's sort of fun to play around with your own language.

For starters, make something simple and build on top of it. First program in your language should probably be something like Hello World. So think of how that program will look like in your language and create an interpreter for just that simple case.

Next maybe augment it with simple formula processing. Conditional statements (If at the very least, else if and else as optional) and a basic loop (loop with break, or for() if you are feeling brave.) Just playing around with this will give you an idea at some of the complexity a script can end up being.

Pure interpreter or compiling to p-code (intermediate op code for a virtual machine) and then interpret. There are pros and cons to either. You should probably play around with both to see which is easier. It can greatly depend on what your language looks like.

If going to p-code create an interpreter for it first and define the p-code. This will be some sort of virtual assembly so very low level calls. Write your programs first in this p-code (or ASCII form of the p-code if your p-code is binary.) to figure out how things should be organized and called. A lot of virtual machines are simple 'stack' based, not like the computer they are running on which makes heavy use of registers in the computer. So much different compiling with register stuff which will add to the complexity of the p-code.

If this is the first time then keep it as simple as possible.

Once you have a p-code interpreter you can work on your compiler of your language. Again, depending on how complex it is you may be able to hand code it or you might need to take a look at some helper tools like lex (GNU flex) and yacc (GNU Bison). Lex is far easier to work with initially and may speed up the tokenization of your language rather than having to write the tokenizing yourself. Yacc is a far more complex beast and for starters you may want to just think about the parsing of the script and dumping out the p-code through unravelling the tokens yourself instead of using this tool. It may actually help you understand what yacc does in the end and appreciate oh so much more.

A pure interpreter is basically a specialized compiler. You still have to tokenize and parse the code but also interpret it. Again depending on the language, this can be very hard or marginally challenging. I won't say interpretting or compiling is simple or trivial.

A good book on the subject would be the dragon book. "Compilers: Principles, Techniques and Tools" by Aho, Sethi an Ullman. amazon.com

Again, start simple. Maybe make a beginner language that looks like a sort of assembly language or a Basic or LISP like. Your brain matter will thank you.
Quote this message in a reply
Member
Posts: 198
Joined: 2005.01
Post: #12
The interesting thing about compilers is that the more "complex" you make it, up to a point, the simpler it actually is. Seems counter-intuitive, but if you e.g. write a thing that reads one line of input at a time and special case parses it, it's actually very difficult to do things like loops. If you start out with the assumption that a program is really a lexical tree structure and go from there, then a great deal of things like yacc suddenly seem pretty simple (it just helps you go from an ocean of tokens to that tree). Lisp also takes on a new level of meaningfulness Smile (GCC actually uses a Lisp type representation internally, last time I checked, and its optimizers work with that.) Then generating the linear pcode is pretty close to just doing a tree traversal. Kinda interesting.

But yeah, starting with something very simple at first is probably the easiest way to get your feet wet...

Cryptic Allusion Games / Cryptic Allusion, LLC
http://www.cagames.com/
Quote this message in a reply
Member
Posts: 131
Joined: 2004.10
Post: #13
True enough but for someone getting their feet wet it may be beneficial to progressively get there. To go from hard coded recursive descent to table driven LALR() from a generator needs some exploration.

Also the book I mentioned may be more textbooky (It is often used in the university classes indicated below) than is desired and from some other discussions I've heard, "Introduction to Compiler Construction in UNIX" may be better as it goes through a pretty decent example using lex and yacc, not a trivial one like a reverse polish notation calculator. Unfortunately this book looks like it's been out of print for a while and I personally haven't actually read it so your mileage may vary.

Just a point of interest maybe, Language processors (compilers, interpreters) is a 4th year and grad level course in university. This is just to give you an idea on the potential difficulty and scope of compilers and interpreters. But again, don't let this disuade anyone from tackling it.
Quote this message in a reply
Member
Posts: 304
Joined: 2002.04
Post: #14
Thanks everyone for all the great responses (even those that warned me off the project which I *knew* was coming)

backslash - text adventure / IF parsers are also difficult and interesting and something Id like to tackle sometime too!

igame3D - oh yeah - I have antack and dead days somewhere - Im going to go look for it. Thanks.

Zekaric and zKing - good advice - Im going to look for those books - its amazing how expensive some comp books are tho.

AnotherJake - I found the Game Scripting Mastery book on safari.oreilly.com and got the online version. Unfortunately it doesnt include the code from the cd which is disappointing. Im looking through it now.

from Zekaric - "Maybe make a beginner language that looks like a sort of assembly language or a Basic or LISP like." - or maaaybe a RPN scripting language. Users wouldnt hate that too much. EMC2^*=

a BASIC syntax would simplify things (and give me warm memories of the ][+ and AppleSoft) so Im considering it.

thanks!
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #15
codemattic Wrote:a BASIC syntax would simplify things (and give me warm memories of the ][+ and AppleSoft) so Im considering it.
Thats what I REALLY wanted for iGame3D but Tobi had never
used BASIC, ever, he went from Hypercard to PowerPC Assembler, and so thats
why our scripting language looks the way it does. I totally wanted to relive my childhood, but I guess so did Tobi and his childhood was only six or less years removed from his age when I met him, compared to my twenty something years.

If you could manage to turn T3DScript to BASIC that would be freakin awesome, since we went from ASSembler-ish to LUA in about a month, I don't see why we can't find a way to support more than one scripting language, would be a very interesting experiment at the least, and then you could have a 3D Modeller and level editor just about ready to go for your projects!
If I knew my way around C, I would help you with the details.

Here's Dead Days Source so you don't have to go looking for it. Looks like there is about 136 command keywords in the T3D_Scripts.h file.
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Suggested Scripting Language Talyn 11 5,337 Dec 31, 2008 03:26 PM
Last Post: Carlos Camacho
  Scripting Language Woes (Lua) Bachus 2 3,777 Oct 14, 2008 09:48 AM
Last Post: Blacktiger
  Looking for a compilable scripting language BinarySpike 8 5,482 Apr 24, 2007 09:59 AM
Last Post: BinarySpike
  Scripting Language: Rolling Your Own Emehr 14 5,600 Feb 10, 2007 08:14 AM
Last Post: IBethune
  Scripting language recommendation? WhatMeWorry 6 3,450 Nov 11, 2006 10:09 AM
Last Post: Duane