1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Public Stats API

Discussion in 'Developer Work and Blog posts' started by Undone, May 9, 2017.

Thread Status:
Not open for further replies.
  1. Lord Andernut

    Lord Andernut Avatar

    Messages:
    3,340
    Likes Received:
    10,087
    Trophy Points:
    165
    Gender:
    Male
    Location:
    New Britannian Market
    I just to clarify... I want a heatmap of MY POT. This would be a useful tool for me to see what areas are not getting foot-traffic and make adjustments for it, and also realize where the organic "hotspots" are with foot-traffic or where people are going out of their way to visit a specific vendor. And hey, do half the people stop walking once they hit the airship? etc.

    I'm not really a fan of all the information the API hands out, but I would still like to use it as a tool for improving my town if possible. (Whether it requires a password or no.)
     
    LoneStranger likes this.
  2. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    Ok, so to clarify a few points of misconception.
    Delay is not about knowing what's there or not.
    Delay is designed to make time-sensitive information less useful.
    Right now there is NO DELAY. In fact, because of this, if you are hellbent on figuring out who is pvping, you can in real time, and I feel this is a great injustice. It takes some work, but it's doable. I did a proof of concept when the API launched to show exactly this data to try to prove my point, but alas it fell on deaf ears but I feel the risk of real-time pvp knowledge was not ranked as highly as i would have hoped in terms of its negative pvp implications by the general consensus and not singling out just Undone, who was helpful in hearing me out and trying to find usable solutions, many of which were put in that mitigated but not removed the risk.

    SotaSTATS incorporates a 15 minute-2 hour delay (based on where the scene is in the priority queue) but that is by choice, not due to the API.
    I've advocated for a delay (1hr-24 hours) for most data since the thing was put in. This won't prevent you from knowing what happened (vendor sales, farming places, what scenes are active), but it will prevent you from immediately profiting from data not everyone has access to (buying something the second it's been listed at an erroneous price, pvping people by knowing where they are online right this second, knowing who has what alt accounts, etc).

    Delays have a purpose.


    @Undone - Needless tag, but a reminder on my stance since vendor API data has been talked about but isn't in yet but has been said to be going in. If vendor sales are put up, they NEED a time delay, and I also advocate heavily that only SALES should be in the api, not listings. Force people to travel and see what's for sale where. If the api registers when/where/howmuch an item is put up for sale, there are several exploitable situations due to existing bots that will arise, not to mention players with private data that I can go more into on request.

    As for @devilcult worry about underground/hidden data - SotaSTATS is designed to alleviate the private data issue, by providing data that gives an advantage (such as lottery ticket numbers) to all players, so its not just programmers who benefit. There are more improvements on the way in that vein, as there are still more things that the public should have easy access too. Just one example, but Farming rates is planned, so people can see the supply generation rates and change crops accordingly.

    Also, I throw my voice behind @Lord Andernut for getting POT scenes their own template name. I However, also voice my support for @Baron Drocis Fondorlatos long-standing request that POT owners should have a setting that exempts their POT from any API data collection within their scene, if they so choose.
     
    Last edited: Sep 25, 2017
  3. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    The whole time delay discussion has not fallen on deaf ears. There isn't a magical switch that I can flip that suddenly delays incoming packets of data to a database meant for real time data ingestion and analysis (or I would have done it). Implementing a time delay with our current framework is not trivial; the compromise was rounding the timestamps to 1hr increments and subsequently to 15 minute increments after feedback from the community.

    So yes, you could theoretically detect specific events as they come through the pipe but you have to continuously query which will likely put you up against the throttle.
     
    Tiina Onir and Umuri like this.
  4. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    I feel i should clarify as the previous post may have given the wrong impression to others. I have since made edits up top, but to re-iterate them here:

    Undone has been awesome about this and has literally done all he could and spent many hours looking into alternatives and identified a few ways in the first few weeks the API was live. We had many lively discussions on it. He really has gone above and beyond since the API was put in to work with me and other people who use it to try to mitigate the damage that the data could cause in the wrong hands who wasn't sharing while ensuring we could still provide useful metrics without breaking the game.

    The deaf ears part was only in reference to the general consensus of the real time pvp chart "not being a big risk" in terms of the grand-scale priority assigned to when they give undone the freedom to do delay work, and not that the delay itself fell on deaf ears. I only reiterated it because i worry a similar issue will happen if/when vendor api data gets pushed through if listing data is put in without the delay. The easiest example is insta-buying anything listed at a bad price (10 cotos for 100 instead of 100k) before the person who submitted it has time to realize their mistake.

    It is not trivial to implement the delay and even the simplest solution we found was i think a good 10-20 hours of dev work on his side plus some from atos.
     
    Tiina Onir likes this.
  5. Solazur

    Solazur Avatar

    Messages:
    1,429
    Likes Received:
    3,477
    Trophy Points:
    113
    Gender:
    Male
    Location:
    Pacific NW
    I may be alone in this but I'm against anything that shows what stuff is selling for.
     
    Elnoth likes this.
  6. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
    I'd like to understand why you are against that.

    Personally I'm interested in the buy/sell prices because I want make sure I'm not getting ripped off and that I am not ripping off my customers.
     
    Sir Aular Glorthalion likes this.
  7. Sir Aular Glorthalion

    Sir Aular Glorthalion Avatar

    Messages:
    97
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Ann Arbor, MI
    I'm behind on this conversation, but I've now skimmed through the 14 pages (oof) of text and I've seen this asked a few times, but I don't see that it was really answered.

    Is there a data dictionary some place that gives a better (English) explanation of what the various functions mean? Some of them seem self-explanatory, but I've been working with data for 12+ years and I know what developers call things and what business people call things can differ wildly ;)

    Functions that I'd specifically like more clarity around:
    • ItemGained_CrownMerchant - is this event triggered when a Crown Merchant has a new item added to their for-sale list?
    • ItemGained_ExplodeItem_Merchant - what in-game action triggers this event?
    • ItemGained_World - How does the 'world' gain an item? What in-game action triggers this event?
    • ItemGained_Merchant - is this triggered when a new item is listed for sale on a public merchant?
    • ItemGained_Merchant - why are there two of these listed?
    • ItemGained_World - why are there two of these listed?
    • ItemDestroyed_Merchant - is this triggered when an item on a merchant is sold?
    Second question:
    For the X,Y,Z coordinate system, how are these anchored/recorded? It doesn't appear to be anchored to a global 0,0 world coordinate; Ardoris should be SW of Soltown and Owlshead far north of both, but I can't get them in that orientation with any combination of XYZ as lat, long, alt:

    [​IMG]
     
  8. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    Where did you get the XYZ coordinates? If you got them from the item event stuff above, I would guess they are relative to the scene's coordinate system, which means they have no bearing on each other. I believe XZ is the real plotting of the coordinates, but some scenes are actually rotated 90 or 180 degrees, so the top of the image isn't always north.
     
  9. Sir Aular Glorthalion

    Sir Aular Glorthalion Avatar

    Messages:
    97
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Ann Arbor, MI
    They're a data point in the JSON file I retrieved from the elasticsearch:

    {
    "_index": "logstash-2017.09.27",
    "_type": "logs",
    "_id": "AV7FwH-qn7c6aq0AJXk5",
    "_score": 5.675345,
    "_source": {
    "ypos": "36.002",
    "Archetype": "Ingredients/Tools/ItemTool_Harvesting_SkinningKnife01",
    "ItemId": "59cc397aabb74c0e58d30ae5",
    "SceneName": "POT_grassland_metropolis_01_template",
    "PricePerUnit": "80",
    "GoldValue": "32",
    "LocationEvent": "ItemGained_Merchant",
    "timestamp": "Sep 27 18:45:00",
    "zpos": "-162.521",
    "xpos": "47.64",

    "Quantity": "1",
    "EconomyGoldDelta": "-48",
    "@timestamp": "2017-09-27T23:45:00.000Z",
    "Price": "80"
    }

    I assumed they were map coordinates for the vendor, but I can't get them to correlate with the actual world map in any way. If they're anchored to each scene, even with rotation, I can build a normalizing table (rotation factor, world X, world Y), join that in, and then plot the data in the world. That seems like a lot of work to plot out, though, especially with the generic scene names.
     
  10. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    I made an example of that with some maps on http://monstersterritory.com/sota
    Not only the map are rotated, but the 0,0 arent in the same place everytime. Mean you can have a map with the 0,0 in center, other one will be top corner.. etc
    I remember adding a variable named "Mirror".. cant find the file right now but that might be because ifound out the map are rotated only in 2 possible way. Older map (R4-) are different than newer one.
    I know i had to go in every map to find out the 0,0 coordination on the map itself (the map u get from the game minimap) then set that mirror variable to 0 or 1 and map would show correctly.. I was using 300x300 pixel map which fitted most of the map except some dungeon, which again was a bit more hardcore cause of different height position (tunnels and such)...
    Anyway that was a long work and i kinda abandonned the project because i need all the mini map image and copy pasting it from the game isnt the best for 200+ map. Also going trough all the map of grundvald to find the position then made it avaible by code took me almost a whole day. Good luck :p
     
  11. Sir Aular Glorthalion

    Sir Aular Glorthalion Avatar

    Messages:
    97
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Ann Arbor, MI
    That seems very strange from a data standpoint. Undone, are there plans to standardize the mapping coordinates in a future release of this? Could there be?
     
  12. Barugon

    Barugon Avatar

    Messages:
    15,679
    Likes Received:
    24,293
    Trophy Points:
    153
    Gender:
    Male
    That's because it's in a spherical Mercator projection and not WGS84.

    Not really. It's probably because each map has it own local coordinate system, so the coordinates you get inside say Ardoris are only applicable there and not on the world map.
     
  13. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Each scene is independent in terms of coordinates and the center point / rotation of each scene is up to the designer.
     
  14. Sir Aular Glorthalion

    Sir Aular Glorthalion Avatar

    Messages:
    97
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Ann Arbor, MI
    That makes the XYZ component of the data feed very cumbersome to use, and the data point to not have much meaning. Not without a translation table to sit between the data points and the overall world map. Is that being kept some place, or documentation about how each scene is oriented? As another posted pointed out, the only other way to see where in the overall world something sits, then, is to go to each scene manually and determine the designer's rotation, center point anchor, etc. Which defeats the purpose of automated data.

    Another thought: Could you expose or include the scene's coordinates in the overworld map?

    I know I'm posting nit-picky kind of comments and suggestions here - overall this is an awesome service and idea. Just trying to understand what's possible and make suggestions that could help people build tools for the community.
     
  15. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    Just a question... why would you want to map individual points of interest within scenes to an overall world map? A relationship between scene POIs and the overland map doesn't exist. This isn't like Ultima 6 or later games where your approach to the location or the location size matters.
     
    Alley Oop likes this.
  16. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    The XYZ coordinate should simply be removed. I suggested that a while back but people wanted it so much (Why? if its not to track stuff that people simply playing the game cant have access)
    They are incoherent if a massive effort isnt done, only programmer with ton of time to waste can actually use that thing. This make it unfair to anyone else.
    You can guess exactly where stuff has dropped and by who, you know what path people take, etc.
    This is in no way usefull for your own self without having an eye on others spying them.
    I once again suggest to hide this data from people. Make it fair for everyone.
     
  17. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    As mentioned by @LoneStranger there is no direct translation between overworld and scene coordinates. Jakub from sotamap did most of the coordinate translations and rotations manually on his end so it is certainly possible on a scene-by-scene basis. But we could probably give you a clearer direction if you elaborated on what you wanted to accomplish.

    The use case I had in mind was scene specific heat maps of recent activity; I personally find these sorts of visualizations very interesting. And I'm not sure how they would provide anyone with a clear advantage in having access to them other than getting a feel for where things happen.
     
    Tiina Onir likes this.
  18. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    Luckily a programmer wasted a ton of time to provide exactly that to everyone!
    SotaSTATS Heatmaps.
    Example shows gold ore drops in elysium mines within a 48 hour window
     
  19. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    The clear advantage they have is they dont have to explore or anything to find stuff. Just check out each maps, see what drops and where...Easy like 1-2-3...
    Just like they do with the recipe, know exactly what vendor sell em even if you try to hide em it will take an hour and the people who have access to that will know exactly where to search... Why even try to hide it...
    Maybe its just me that see it this way i dunnoh... Where is the adventure if the knowledge come from a 3rd party program.

    I guess its the same having a wiki or anything, just its not realtime like this and you still have to search for what you want. Not just select a map and check all the loot that drop in that map at once.
     
  20. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Rainbow color maps by default? Shame on you!
     
    hroth and Umuri like this.
Thread Status:
Not open for further replies.