View Single Post
  #2    
Old July 27th, 2011 (10:35 PM).
Missingyep Missingyep is offline
     
    Join Date: Jul 2010
    Posts: 275
    Improved Eval (Advanced Tutorial)
    I've since improved the eval technique. Here is the revised procedure for creating an eval script. Bear in mind that this is quite complex. (At this point, I'm wondering whether documentation of my eval technique would be more fitting in Tutorials or in R&D...)

    With these new improvements, the technique has considerably increased in complexity, but this increase allows for larger and more stable dynamic scripts.

    Explanation
    We start by writing our dynamic script -- the dynamic code that we are running with this technique. This is the script that we compile at run-time. We write hex bytes to a range of variables starting at 0x4100. You can use a different range of variables, but you will have to alter some of the math I do with RAM offsets (good luck figuring it out, I only managed to finish it with some shotgun debugging).

    Here is where we run into the first implementation problem, which -- through lots of searching and sheer dumb luck -- I was able to solve quickly and elegantly. While "meta" variables like 0x800D are stored at fixed RAM locations, all other variables -- including the ones we will be using for our dynamic script -- are DMA-protected.

    DMA, if memory serves, stands for "Dynamic Memory Access". It is a technique used to stop people from GameSharking OVER 9000 Master Balls into their Bag. DMA-protected information is moved to a random RAM offset, and the location of this offset is stored in a fixed RAM location. For FireRed, this relocation happens whenever the screen changes (i.e. opening a PokeMart, entering a battle, opening the bag, warping), and the fixed RAM location (which holds the offset currently in use by DMA) is at 0x3005008. DMA-protected information includes Bag data, all script flags, and all script variables (except for "meta" variables like 0x800D).

    We need to take the four bytes at that location and reverse them. This gives us the current RAM offset of the start of all DMA-protected data; the base DMA offset. Adding 0x1000 to the base DMA offset gets us the location of variable 0x4000. Ergo adding 0x1200 to the base DMA offset gets us the location of variable 0x4100.

    Therefore in order to call our script, we must take the base DMA offset, add 0x1200 to it, and then run a "call" command at that offset. Here, we run into our second implementation problem: "call" does not accept a dynamic value (i.e. two variables). A "call" command can only call the address specified when its containing script is compiled.

    We work around this by compiling an auxiliary script at run-time into the RAM used to store variables 0x800D and onward. This is our dynamic loading script. The effect is that we compile a very small, and (through sheer dumb luck and good fortune) unlikely-to-be-corrupted caller script into the RAM used for 0x800D. We call that script, which in turn locates and calls the larger (or not, but it certainly can be) dynamic script that we have stored in variables 0x4100 and onward.

    The execution of all code hence occurs in the following manner:
    • Master script writes hex values to variable 0x4100 and onward; this is the compilation of the dynamic script. Master script then calls the loading helper.
    • The loading helper finds the current RAM offset of 0x4100. It compiles the dynamic loading script, which consists of a "call" to that offset followed by a "return".
    • The loading script calls the dynamic script.
    • The dynamic script executes and returns. Execution returns to the loading script.
    • The loading script returns. Execution returns to the master script.

    Review of terminology
    The master script is the total XSE script that you write. It compiles the dynamic script and calls the loading helper.

    The dynamic script is a script that is compiled at run-time, by storing hex values in variables 0x4100 and onward.

    The loading helper is a piece of script code that finds the current location of the frequently-moved variable RAM. It compiles the loading script, which calls the dynamic script.

    The loading script is a script that is compiled at run-time, by storing hex values in variables 0x800D through 0x8010 (as well as the nonexistent variable 0x800E). We also do some math with variables 0x8011.

    The loading helper and loading script
    If you're running a FireRed BPRE v1 US ROM, then I've already done all the work here for you. Take a look at this:

    Code:
    #org @fExecuteDynamicScript
    /*
     Compilation (loading script):
    */
    writebytetooffset 0x04 0x020370D0 // 0x800D: "call", address byte
    copybyte 0x020370D1 0x03005008
    copybyte 0x020370D2 0x03005009 // 0x800E: address byte, address byte
    copybyte 0x020370D3 0x0300500A
    copybyte 0x020370D8 0x020370D2 // math on 0x800E (address += 0x1200)
    copybyte 0x020370D9 0x020370D3
    addvar 0x8011 0x0012
    copybyte 0x020370D2 0x020370D8
    copybyte 0x020370D3 0x020370D9
    copybyte 0x020370D4 0x0300500B // 0x800F: address byte, "return"
    writebytetooffset 0x03 0x020370D5
    writebytetooffset 0xFF 0x020370D6 // 0x8010: script terminating byte, nop
    writebytetooffset 0x00 0x020370D7
    /*
     Execution (loading script -> dynamic script):
    */
    call 0x020370D0
    return
    That is the loading helper, which contains code that compiles and executes the loading script.

    You'll notice that we use 0x8011 to do math on 0x800E indirectly. The reason that we cannot directly manipulate 0x800E is because it is not actually a variable. Setvar, addvar, subvar, and pals refuse to operate on it. If you try to copyvar the value of 0x800E into some other variable, the game will instead copy some random word from Darwin-only-knows-where.

    Here is where we hit that stroke of dumb luck I mentioned earlier -- the one that prevents corruption of the loading script during execution of the dynamic script. The "return" byte in the loading script is contained in the RAM that would be used for 0x800E if 0x800E existed... But since 0x800E doesn't exist, there's no risk of [set|add|sub|copy]var statements in the dynamic script corrupting it! Unless the dynamic script deliberately uses writebytetooffset/copybyte/etc. to overwrite 0x800E's RAM, there's no risk of the loading script failing to return due to it being overwritten.

    (Of course, overwriting 0x8010 -- the var containing the FF byte that terminates the script -- might be problematic. Unless XSE just adds that as padding, and it doesn't actually do anything. In any case, I haven't tested this, so do be careful with 0x8010.)

    In any case, all you have to do is copy that routine into your script. You actually only need to compile it once; so long as you're careful to use #remove and not #removeall, you can manually reference that one copy of the routine from any and all scripts that use this technique (so long as you write down what offset the routine is written to).

    A sample dynamic script
    Here is an example script. It generates a random Pokemon index, and then shows the image of that Pokemon. As a bonus, it will also show a dialog box listing the name and native region of that Pokemon. The full code, spoiler'd for your convenience:

    Spoiler:
    Code:
    #dynamic 0x800000
    
    #org @start
    lock
    faceplayer
    /*
     Get a random species number.
    */
    random 0x0181
    copyvar 0x4000 0x800D
    addvar 0x4000 0x0001
    compare 0x4000 0x00FB
    if 0x02 call @adjustHoennNumber
    
    /*
     Compiling dynamic script.
     Script:
        showpokepic XXYY 0x00 0x00
        bufferpokemon 0x00 XXYY
        return
     Hex:
        00 75|YY XX|00 00|
        7D 00|YY XX|
        03 FF
     Vars:
        7500|XXYY|0000|007D|XXYY|FF03
    */
    setvar 0x4100 0x7500
    copyvar 0x4101 0x4000
    setvar 0x4102 0x0000
    setvar 0x4103 0x007D
    copyvar 0x4104 0x4000
    setvar 0x4105 0xFF03
    
    msgbox @sAnnounce 0x02
    call 0x800EC4 // dynamic script loader and executer
    call @checkRegion
    msgbox @sExplain 0x02
    release
    hidepokepic
    end
    
    /*
     There are 0x1A "?" Pokemon between the 
     last Johto and first Hoenn Pokemon. If 
     our random number selected a Hoenn 
     species, we need to adjust for this.
    */
    #org @adjustHoennNumber
    addvar 0x4000 0x001A
    return
    
    /*
     Check the Pokemon index number to find 
     what region it is from. Most of the 
     Pokemon are in Pokedex order already, 
     which makes this easy for us.
    */
    #org @checkRegion
    call @bZALGO
    compare 0x4000 0x0001
    if 0x4 call @bKanto
    compare 0x4000 0x0098
    if 0x4 call @bJohto
    compare 0x4000 0x00FC
    if 0x4 call @bZALGO
    compare 0x4000 0x0115
    if 0x4 call @bHoenn
    compare 0x4000 0x019C
    if 0x4 call @bZALGO
    return
    
    #org @bKanto
    bufferstring 0x01 @sKanto
    return
    #org @bJohto
    bufferstring 0x01 @sJohto
    return
    #org @bHoenn
    bufferstring 0x01 @sHoenn
    return
    #org @bZALGO
    bufferstring 0x01 @sZALGO
    return
    
    #org @sAnnounce
    = Hey, look at this!
    #org @sKanto
    = Kanto
    #org @sJohto
    = Johto
    #org @sHoenn
    = Hoenn
    #org @sZALGO
    = ZALGO
    #org @sExplain
    = It's a [buffer1], from the [buffer2]\nregion.


    Let's look at this code piece-by-piece.

    Code:
    #dynamic 0x800000
    
    #org @start
    lock
    faceplayer
    /*
     Get a random species number.
    */
    random 0x0181
    copyvar 0x4000 0x800D
    addvar 0x4000 0x0001
    compare 0x4000 0x00FB
    if 0x02 call @adjustHoennNumber
    This is simple enough. We choose a random species number. If it is above a certain value, we add 0x1A to it (@adjustHoennNumber) to account for the several unused "?" Pokemon that lie between the Johto and Hoenn Pokemon indices.

    Code:
    /*
     Compiling dynamic script.
     Script:
        showpokepic XXYY 0x00 0x00
        bufferpokemon 0x00 XXYY
        return
     Hex:
        00 75|YY XX|00 00|
        7D 00|YY XX|
        03 FF
     Vars:
        7500|XXYY|0000|007D|XXYY|FF03
    */
    setvar 0x4100 0x7500
    copyvar 0x4101 0x4000
    setvar 0x4102 0x0000
    setvar 0x4103 0x007D
    copyvar 0x4104 0x4000
    setvar 0x4105 0xFF03
    Here, we compile our dynamic script. You'll see that I've translated the script into hex. Doing so is fairly simple:
    • XSE's Command Help (F1) tells you the hex byte of the command itself.
    • If a command takes a multibyte value, reverse the byte order when translating to hex.
    • When translating from hex to vars, group bytes into words and then reverse the bytes within each word.
    • Strategically-placed nops (0x00) before a command can save you some tricky math. Above, I used one nop so that XXYY remained as one whole variable, rather than being split across them.
    Once we've translated the script into vars with hex values, we simply write them into our specially-designed eval variables, starting with 0x4100. It's a simple series of setvars, plus a copyvar or two for whichever parts of our script are actually dynamic. (In this case, the Pokemon species index, represented by "XXYY" in the code comments.)

    Moving on:

    Code:
    msgbox @sAnnounce 0x02
    call 0x800EC4 // dynamic script loader and executer
    And here's where stuff goes down. We show a simple message box, and then we call our loading helper. 0x800EC4 is not the offset you use in your ROM. You use the offset that @fExecuteDynamicScript was compiled to in the previous section ("The loading helper and loading script"). 0x800EC4 just happens to be the offset of that routine in my ROM hack.

    Anyway, that call triggers the lo-- wait, I've already explained this, like, three times, haven't I? :P

    The rest of the script is just cleanup and message displays. We've covered everything that's important.

    Case study: simulating arrays using dynamic scripts
    This case study requires JPAN's FireRed Hacked Engine.

    Consider this code:
    Spoiler:
    Code:
    /*
     Demo:
      - Selection from arrays
         - Using user-entered multichoice value with 
           dynamic script
         - Using random value with dynamic script
    */
    
    #dynamic 0x800000
    
    #org @start
    lock
    faceplayer
    msgbox @sIntro 0x02
    
    /*
     Constructing our makeshift array:
    */
    setvar 0x4000 0x0117 // Song: Cinnabar Island
    setvar 0x4001 0x0120 // Song: Mt. Moon
    setvar 0x4002 0x0132 // Song: Pokemon Tower
    setvar 0x4003 0x0151 // Song: Sevii Islands 6 and 7 / Violet City
    setvar 0x4004 0x0157 // Song: Deoxys Encounter
    
    /*
     Asking the user to pick an option:
    */
    setvar 0x8006 0x0 // Multichoice 0
    loadpointer 0x0 @sSong0
    special 0x25
    setvar 0x8006 0x1 // Multichoice 1
    loadpointer 0x0 @sSong1
    special 0x25
    setvar 0x8006 0x2 // Multichoice 2
    loadpointer 0x0 @sSong2
    special 0x25
    setvar 0x8006 0x3 // Multichoice 3
    loadpointer 0x0 @sSong3
    special 0x25
    setvar 0x8006 0x4 // Multichoice 4
    loadpointer 0x0 @sSong4
    special 0x25
    setvar 0x8006 0x5 // Multichoice 5
    loadpointer 0x0 @sRandom
    special 0x25
    setvar 0x8006 0x6 // Multichoice 6
    loadpointer 0x0 @sCancel
    special 0x25
    multichoice 0x0 0x0 0x25 0x0 // Multichoice trigger
    copyvar 0x4010 0x800D // save result
    compare 0x4010 0x0005
    if 0x1 call @randomize
    compare 0x4010 0x0006
    if 0x1 call @cancel
    
    /*
     Compiling dynamic script.
     Description:
        Copies the value of a variable into 
        another variable. The variable to be 
        copied is dynamically selected at 
        run-time. Basically, we can pick an 
        element out of an "array" using a 
        variable index, and return that 
        element by copying it into variable 
        0x4011.
     Script:
        copyvar 0x4011 0xXXYY
        return
     Hex:
        00 19|11 40|YY XX|
        03 FF
     Vars:
        1900 4011 XXYY FF03
    */
    setvar 0x4100 0x1900
    setvar 0x4101 0x4011 // Destination variable for returning the element
    copyvar 0x4102 0x4010 // Variable holding the index of the element we want
    addvar 0x4102 0x4000 // The array starts at variable 0x4000, so add that amount to the zero-based index to get a variable ID.
    setvar 0x4103 0xFF03
    
    call 0x800EC4 // dynamic script loader and executer
    
    /*
     Compiling dynamic script.
     Description:
        Selects a music track based on a 
        value we inject.
     Script:
        playsong 0xXXYY 0x00
        return
     Hex:
        00 33|YY XX|00 03|FF 00
     Vars:
        3300 XXYY 0300 00FF
    */
    setvar 0x4100 0x3300
    copyvar 0x4101 0x4011
    setvar 0x4102 0x0300
    setvar 0x4103 0x00FF
    
    call 0x800EC4 // dynamic script loader and executer
    
    msgbox @sSongChanged 0x02
    
    release
    end
    
    #org @randomize
    random 0x0005
    copyvar 0x4010 0x800D
    return
    
    #org @cancel
    release
    end
    
    #org @sIntro
    = Hi, there! Would you like to change\nthe music?
    #org @sSong0
    = Cinnibar Theme
    #org @sSong1
    = Mt. Moon Theme
    #org @sSong2
    = Pokemon Tower Theme
    #org @sSong3
    = Violet Theme
    #org @sSong4
    = Deoxys OW Theme
    #org @sRandom
    = Surprise me
    #org @sCancel
    = I'd rather not
    #org @sSongChanged
    = Lemme know if you wanna change the\nmusic again.


    This master script powers an NPC that lets you choose from a menu of music tracks. Pick one, and he'll change the currently-playing song to it. Alternatively, you can have him pick one for you at random, or you can decline and not change the music at all.

    Let's step through it to see some of the techniques it uses, which can of course be generalized and used in other applications.

    Code:
    /*
     Constructing our makeshift array:
    */
    setvar 0x4000 0x0117 // Song: Cinnabar Island
    setvar 0x4001 0x0120 // Song: Mt. Moon
    setvar 0x4002 0x0132 // Song: Pokemon Tower
    setvar 0x4003 0x0151 // Song: Sevii Islands 6 and 7 / Violet City
    setvar 0x4004 0x0157 // Song: Deoxys Encounter
    Here, we are creating a makeshift array -- in this case, of song indices -- by writing to a sequence of consecutive variables. We choose 0x4000 for convenience sake, but any variables (except 0x4100 and onward!) could be used for the purpose.

    That's all an array really is -- a list of things. In this case, we're just spreading that list across multiple variables. But how do we read from it dynamically?

    Code:
    /*
     Compiling dynamic script.
     Description:
        Copies the value of a variable into 
        another variable. The variable to be 
        copied is dynamically selected at 
        run-time. Basically, we can pick an 
        element out of an "array" using a 
        variable index, and return that 
        element by copying it into variable 
        0x4011.
     Script:
        copyvar 0x4011 0xXXYY
        return
     Hex:
        00 19|11 40|YY XX|
        03 FF
     Vars:
        1900 4011 XXYY FF03
    */
    setvar 0x4100 0x1900
    setvar 0x4101 0x4011 // Destination variable for returning the element
    copyvar 0x4102 0x4010 // Variable holding the index of the element we want
    addvar 0x4102 0x4000 // The array starts at variable 0x4000, so add that amount to the zero-based index to get a variable ID.
    setvar 0x4103 0xFF03
    That's how.

    We compile a dynamic script that will take some variable XXYY, and copy its value into 0x4011. Instead of using separate code branches for each possible selection, we use a dynamic script and a "pseudo-array".

    That script runs:
    Code:
    call 0x800EC4 // dynamic script loader and executer
    And then we use 0x4011 to pick the music track. We do that with another script, since (to the best of my knowledge) we can only pass raw values, not variables, into playsong.

    Code:
    /*
     Compiling dynamic script.
     Description:
        Selects a music track based on a 
        value we inject.
     Script:
        playsong 0xXXYY 0x00
        return
     Hex:
        00 33|YY XX|00 03|FF 00
     Vars:
        3300 XXYY 0300 00FF
    */
    setvar 0x4100 0x3300
    copyvar 0x4101 0x4011
    setvar 0x4102 0x0300
    setvar 0x4103 0x00FF
    
    call 0x800EC4 // dynamic script loader and executer
    And there we go.

    This may seem like total overkill. Why not use code branches for each selection, instead? Well, if you thoroughly understand this technique, then you may find that these two benefits are worth the added complexity (it really depends on your tastes as a programmer):
    • Ditch all the comments, and (in terms of line count) this code is shorter than an equivalent with branches would be. The more options you have, the bigger the size difference becomes.
    • In this code, your array of songs is right at the top. You don't need to sift through instructions and branches to change the songs; they're right up in an obvious and prominent spot, and easily alterable.

    Conclusion
    This is how you can dynamically compile a script of (almost) any size at run-time, without the risk of that script overwriting itself when its instructions are processed.

    As opposed to my first (prototypical) eval method, in the preceding post. That method limits the size of the script to the RAM block reserved for "meta" variables (a rather small area), and runs the risk that the script might overwrite itself (i.e. one of its commands sets 0x800D, nuking the first two bytes of the script) in a manner that could cause major problems (i.e. one of the later "meta" variables gets overwritten and boom, your script no longer returns or ends).

    The only thing to keep in mind is that, as with the old method, you will nuke the "meta" variables when running dynamic scripts (specifically, when the loading script is compiled). If you need things like LASTTALKED or PLAYERFACING, copy them to standard variables before calling the loading helper.

    Complicated? Yes. Overkill? Most definitely. But useful? Absolutely.
    Reply With Quote