1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Public Stats API

Discussion in 'Developer Work and Blog posts' started by Undone, May 9, 2017.

Thread Status:
Not open for further replies.
  1. Whitestag

    Whitestag Avatar

    Messages:
    197
    Likes Received:
    250
    Trophy Points:
    18
    My question is.

    Hey port ya see anything funny going on with people that have dev access and pretesting access in terms of bugs or exploits
    that are utilized to the max up until they are corrected ?? o_O

    I respect port a lot more for opening this up to players to view logs. :)

    Do not let the fox build the hen house ... :cool:
     
  2. Daxxe Diggler

    Daxxe Diggler Avatar

    Messages:
    2,692
    Likes Received:
    5,711
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Virtue Oasis - Hidden Vale
    I like your simple queries setup. Very easy to use and good descriptions for everything.

    However, I tried the Pwned by Avara one and noticed the data for some of the rows were displayed in the wrong columns. Not sure if your table is messed up or if the data came in wrong?

    Besides that, I saw my name on there and I haven't opted in to this yet. How is that possible?

    And to make things worse, the one time I got killed by him... It happened like 6 times within a few minutes ( I was stubborn and kept trying to run by him to get to the other side of the zone). Shouldn't I be listed more than once?
     
  3. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    yeah thats because they change the timestamp on the server to 15 minute instead of hours. Ill put it back at old view to have headers on every rows. Was trying to make it more beautifull but guess it will have to wait a bit :p
    there was also a problem while listing non specific thing like silver ore as archetype. it can give me the lootgenerated, or item destroyed, so the table come weird like you saw them if its not a specific search :\

    EDIT: I added a checkbox so you can switch the view of table with or without header every rows.
     
    Last edited: May 19, 2017
  4. Titania Xylia

    Titania Xylia Avatar

    Messages:
    398
    Likes Received:
    1,212
    Trophy Points:
    43
    Gender:
    Male
    I apologize if this has been asked before, but would it be possible to add PlayerName to the Item events?

    Also, can we send multiple SceneName's in one query?

    Thanks! I'm enjoying the system, so far. It's been fun.
     
  5. Black Tortoise

    Black Tortoise Avatar

    Messages:
    1,961
    Likes Received:
    3,655
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Storm's Reach
    {nvm ignore me, i was mistaken about my UI... and cant seem to find the delete button for this post}
     
    Last edited: May 20, 2017
  6. Titania Xylia

    Titania Xylia Avatar

    Messages:
    398
    Likes Received:
    1,212
    Trophy Points:
    43
    Gender:
    Male
    It would be great if you had a listing that mapped archetype to the in-game displayname:

    "Consumables/Potions/ItemPotion_ObsidianCapacity": "Obsidian Potion of Capacity"

    The deeper you get into archetypes the more arbitrary the scheme is, so it's tough to come up with a boiler plate conversion between what an item is called in-game versus the API.
     
  7. Phil Scott

    Phil Scott Avatar

    Messages:
    74
    Likes Received:
    41
    Trophy Points:
    20
    Gender:
    Male
    hroth likes this.
  8. Titania Xylia

    Titania Xylia Avatar

    Messages:
    398
    Likes Received:
    1,212
    Trophy Points:
    43
    Gender:
    Male
    What are the rate limits on the API? x/min? x/hr?

    I finally hit it accidentally trying to catch up on my data backlog. I definitely was pushing requests absentmindedly.
     
    that_shawn_guy and Tiina Onir like this.
  9. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    I'd like an answer to this question, too, before I start seriously working with this.
     
  10. Echondas

    Echondas Bug Hunter Bug Moderator

    Messages:
    3,785
    Likes Received:
    4,001
    Trophy Points:
    165
    Gender:
    Male
    Location:
    NY
    Do the stats collected reflect items only picked up by players - or do they also include loot on corpses that was never collected, but 'dropped' (ie. abandoned corpses by players or enemies killed by enemies)

    Also - if not already in place, I think it would be a good idea to collect data on what items are "destroyed" - and cross reference those when making any decisions about altering drop rates, - since I think most people use "E" to loot since clicking item by item takes more time and also leaving corpses increases the respawn time -- and then shortly thereafter "destroy" most of the junk items they picked up that are weighing them down (animal carcasses, non-pristine heads, rusty daggers, shoddy longbow, etc)
     
    Last edited: May 29, 2017
  11. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    I know people who DON'T loot-all specifically because it increases spawn time (that's anecdotal though, I don't know the percentages either way). If you're mining this can be a benefit. I also don't think many destroy things weighing them down, I think far more run back to a house/bank and stockpile and/or vendor trash it.
     
  12. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    The rate limit is approximately 10-12 requests per minute OR approximately 20 megs per 10 minutes.
    Hitting either of those limits will yield the reject. For context, ~10k results generally ~4 mb (+/- .5) if you utilize everything, so basically the limit is 10-12 requests per min and no more than 50k results per 10 minutes.
    Also, whatever was rejected still counts towards your limit.



    Sorry i've been lax on posting, but been out of town more days than in it this month, so my free time to work on this has been slow.
    But SotaSTATs is almost ready for launch, and will give a playground for those who are less tech inclined, as well as (hopefully) severely lighten the load poor Portalarium is facing.
     
    hroth, Tiina Onir and Kabalyero like this.
  13. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    No worries! I'm always glad to see your insights, Umuri!
     
  14. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    is the data coming from the elasticsearch actually compressed? Cause it doesnt seem so and i think it must be done server side... that could reduce alot the bandwith
     
  15. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    I can look into turning on compression today, but I think the client has to put it in their header in order to request gzipped responses.
     
  16. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    That's a good point. It's simple text, so it should compress very well, although the client and server would negotiate that. Here's an example of how to negotiate and handle that in python (I leave sorting out other languages as an exersize to the reader): https://github.com/elastic/elasticsearch-py/issues/252

    The upshot is, if you use something like this in your code, it should still work if compression is on or off. So even if it takes @Undone a while to get that enabled, you're good to go.

    While doing this will likely slow down your response, and probably cause more processing load on the server, it should be easier on bandwidth. Like I said, if your client implements this, Undone can turn on or off compression depending on if the ES setup is bandwidth bound or processing bound.
     
    Last edited: Jun 2, 2017
  17. Dereck Visaard

    Dereck Visaard Avatar

    Messages:
    252
    Likes Received:
    444
    Trophy Points:
    40
    Gender:
    Male
    Location:
    Austin, TX
    @Undone and others, thanks for creating this awesome utility. I'm excited to see it grow. A few questions:

    1) If I want to when any player buys a specific item type in a specific scene, is the ItemGained_Merchant the correct LocationEvent to use?
    2) Can we have a list of the different possible archetypes? They seem quite arbitrary.

    Thanks!
    Ryan
     
  18. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    yep but at less thats gonna reduce the bandwith by like 95% lol

    All ya gotta do (on our side, client side) is write a header saying Accept-Encoding: gzip (in a http header 200 ok)
     
  19. Tiina Onir

    Tiina Onir Avatar

    Messages:
    1,103
    Likes Received:
    1,900
    Trophy Points:
    125
    Location:
    Bramble, South Paladis
    Probably not 95%, but I would expect 40% - 60% for clients the implement it (remember, Undone is going to care more about the aggregate of everyone hitting it, not just one or two clients). Certainly not something to shake a stick at!
     
  20. Jaanelle DeJure

    Jaanelle DeJure Avatar

    Messages:
    1,845
    Likes Received:
    4,276
    Trophy Points:
    113
    I'm sure you know this, but just to point out for the other users that you can use complex boolean queries and wildcards to shorten and weed out responses you don't want. For example:

    LocationEvent: *LootGold* NOT (SceneName: *mine* OR SceneName: *city*) AND GoldValue: 25

    Archetype: *HemlockBorer* AND LocationEvent: ItemGained_Loot* AND (SceneName: *R6* OR SceneName: *blackblade*)

    Archetype: *silver* NOT (LocationEvent: *destroy* OR LocationEvent: *craft*) NOT SceneName: *mines*

    LocationEvent: *Gained_World* AND Archetype: *port* NOT Archetype: *book*
     
    Last edited: Jun 3, 2017
Thread Status:
Not open for further replies.