1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Public Stats API

Discussion in 'Developer Work and Blog posts' started by Undone, May 9, 2017.

Thread Status:
Not open for further replies.
  1. xadoor

    xadoor Avatar

    Messages:
    261
    Likes Received:
    337
    Trophy Points:
    28
    dragon crap in the last 30 days...I kinda think if you divide total #heads by total haunches you are gonna be ballpark droprate:

    1 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Green",
    2 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Blue",
    3 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_White",
    9 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Black",
    15 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Red",
    18 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Orange",
    35 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonHead_Yellow",
    58 "Archetype" : "Books/KnownUltimaDragons",
    105 "Archetype" : "Recipes/Tailoring/Specific_Armor/RecipeArmor_Helm_AnimalDragon",
    254 "Archetype" : "Ingredients/Components_Creature/ItemComponent_DragonNeckBone",
    4614 "Archetype" : "Ingredients/Butchery/ItemRawMat_Carcass_Dragon",
     
    Bom and Elnoth like this.
  2. Hemswal The Descended

    Hemswal The Descended Avatar

    Messages:
    734
    Likes Received:
    1,524
    Trophy Points:
    93
    The spider belt only falls off of the spider queen.

    Also, artifacts are hidden in the API.
     
    Mastrix and Bom like this.
  3. Elnoth

    Elnoth Avatar

    Messages:
    724
    Likes Received:
    1,557
    Trophy Points:
    93
    @xadoor thanks for that. Using your method of haunches divided by heads comes out at 1 in 55, so I guess I have just been unlucky.
     
    Bom and xadoor like this.
  4. xadoor

    xadoor Avatar

    Messages:
    261
    Likes Received:
    337
    Trophy Points:
    28
    yeah that assumes all dragons have the same droprate...no idea of thats true
    also some dragons dont drop any haunces so add a few more to that number

    anyone know how I'd see monster kills? I guess I need to go back and see whats available in that api, i've been using it with curl to search for drops for so long I forgot what else is avail.
     
    Bom likes this.
  5. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Bom and Browncoat Jayson like this.
  6. Browncoat Jayson

    Browncoat Jayson Legend of the Hearth

    Messages:
    6,334
    Likes Received:
    14,098
    Trophy Points:
    153
    Are all of the maps supposed to be listed in that file? As I was looking through some data, I found at least a couple that were not.

    Novia_R7_Forest02_01_Cave
    Novia_R1_Underground01_01
    Novia_R6_Underground03_01

    My guess is these are the Boletusporis Regem cave, the Lost Whiteguard Silver Mines, and the Broken Echoes Silver Mine, based on the content. But this does make me question if there are other scenes missing from the list.

    (Edited to add other missing mine)
     
    Last edited: Mar 7, 2019
    Bom likes this.
  7. Browncoat Jayson

    Browncoat Jayson Legend of the Hearth

    Messages:
    6,334
    Likes Received:
    14,098
    Trophy Points:
    153
    Ok, I've been able to use the API to pull monster kills for various scenes using this command:

    python download_quick.py -o ResoluteSewers.json -tf 30 "MonsterKilledByPlayer AND SceneName:ResoluteSewers"

    However, per the Git notes, I should be able to pull drops the same way, using Loot Generated. So I tried this:

    python download_quick.py -o Blueprints.json -tf 30 "LootGenerated AND ItemId:*Blueprint*"

    This returns zero results. Ok, I probably need to get a more exact ItemId, so I'll run this for all of the drops and see if I can find one

    python download_quick.py -o Loot.json -tf 30 "LootGenerated"

    However, this still get zero results. Looking back at previous pages, it seems like some queries used Archetype:*Blueprint* instead of ItemId, but this didn't work either.

    Is the LootGenerated query broken, or am I doing something incorrect?


    Nevermind. The actual call should be ItemGained_Loot, not LootGenerated like in the GitHub example. Working now.
     
    Last edited: Mar 8, 2019
    Bom and Jaesun like this.
  8. Mastrix

    Mastrix Avatar

    Messages:
    10
    Likes Received:
    16
    Trophy Points:
    3
    Gender:
    Male
    Location:
    New Hampshire, USA
    Hi there! I haven't been active for a while because life happens, but I am going to be looking into a bunch of stuff.
    This thread is quite long. Is anyone able to point me at a place where I can find all of the API documentation that is open to player use?

    Apologies if this seems out of place. I'm just getting my feet wet in the forums here, so if I am making a faux pas, please tell me. Thank you!

    EDIT: To clarify, I am hoping there is more available than the Public Stats API (which is awesome, don't get me wrong). I am just hoping there are more API calls I can configure.
     
    Last edited: May 19, 2019
  9. Cinder Sear

    Cinder Sear Avatar

    Messages:
    2,576
    Likes Received:
    3,836
    Trophy Points:
    165
    Location:
    Spite
    Mastrix likes this.
  10. Mastrix

    Mastrix Avatar

    Messages:
    10
    Likes Received:
    16
    Trophy Points:
    3
    Gender:
    Male
    Location:
    New Hampshire, USA
    Thank you! I appreciate the help.
    If you provide bad data in an API call, you can expose some endpoints and data paths. I'm not sure if those are open, but I'll try and get them to work. I'll let you know how that goes.
    The Sources are useful too. There are some strings with plain text paths on them.

    Basically, I want to make sure I know all of the endpoints I should have access to. Then, I want to try to access the ones I shouldn't be able to, but I'll deal with that once I get there. I don't expect that to work.

    I'd LOVE to see the swagger docs for the whole API, even if I can't authenticate. U'll let ya kn9w what I find.
     
  11. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
    @Mastrix, the "API" is just Elastic Search with POST and some feature disabled. The endpoints are for it are well documented for that project.

    The main downside is that POST request are disabled here. Many ES drivers, modules, etc rely on POST being available. So, it may be a little tricky to get things working depending on the ES support available for your language of choice.
     
    Mastrix likes this.
  12. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
  13. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
    Looks like we got a new "Fishing" LocationEvent with the build 993.

    Code:
          {
            "_index": "logstash-2019.06.14",
            "_type": "doc",
            "_id": "V7djWGsB9DlaFXTYls26",
            "_score": 1.0,
            "_source": {
              "SceneName": "Novia_R6_Volcanic01_02",
              "zpos": "-28.16",
              "LocationEvent": "Fishing",
              "xpos": "14.53",
              "ypos": "65.39",
              "@timestamp": "2019-06-14T23:45:00.000Z",
              "PlayerName": "Thorwalg",
              "timestamp": "Jun 14 18:45:00"
            }
     
    Mastrix, Tiina Onir, Jaesun and 2 others like this.
  14. Hemswal The Descended

    Hemswal The Descended Avatar

    Messages:
    734
    Likes Received:
    1,524
    Trophy Points:
    93
    Neat! Now we can enhance the fishing tab in SotAssist from just showing them as looted items. :)

    Weeee! Give us MOAR API STUFF! ;)
     
  15. Browncoat Jayson

    Browncoat Jayson Legend of the Hearth

    Messages:
    6,334
    Likes Received:
    14,098
    Trophy Points:
    153
    So what do you guys use to parse the json files once they are extracted? I was using a simple json to csv converter, which is fine for 10k records at a time, but not useable for millions of data points across multiple files. Looking for suggestions before I start trying to hack together sometime myself.
     
  16. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
    I'm using mostly Python and Go. Both languages have builtin libs that parse JSON into native objects.

    Normalizing the different events into a single CSV would be difficult. They share some common fields. But, not all fields are in all event types. What are you feeding the data into? CSV always makes me think MS Excel.
     
    Tiina Onir likes this.
  17. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    I'm using python for all the things. It works fantastically for parsing json.

    I disagree that it would be difficult: as long as you're okay with null fields it shouldn't be hard at all. That said, it doesn't seem that useful of a thing to do.

    I'm shoving it (the data directly from the API, not Shawn's copy) into postgres, which has the same issue. ¯\_(ツ)_/¯ I just accept that fields will be null.
     
    Last edited: Jun 15, 2019
  18. Browncoat Jayson

    Browncoat Jayson Legend of the Hearth

    Messages:
    6,334
    Likes Received:
    14,098
    Trophy Points:
    153
    Basically, trying to use it to find uniques for the wiki. I'd be able to update creatures in each scene, for example.

    I'll see how much a pain it is to learn python.
     
  19. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    node js? they have a elastic search library
     
  20. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
    @Undone / @Chris I'd like to request a few thing that need to be added to the API data. Everyone else, I'd also like your input on these.

    1) All questions item drops/awards. There are a few. But, most of the main quests (dread items, shroud, etc) seem to be missing.

    2) All public chat messages. Trade, zone, etc.

    3) If I am a guild leader, and set to share my name in the API, /guildactivitylog (or something specific) should put that information into the API.

    4) Fish details (fish type, size, etc)


    Thank you
     
Thread Status:
Not open for further replies.